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By 

THOMAS  ROBERT  RENCKLY 
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Major  Department:  Instructional  Leadership  and  Support 

This  study  represents  the  collective  observations  of  550 
subjects  in  the  Navy's  Enlisted  Recruiter  Orientation  Course  over 
a one  and  one-half  year  period.  Its  purpose  was  to  determine  the 
effectiveness  of  a spiral-sequenced  (revised)  version  of  the 
course  against  a linear-sequenced  (traditional)  version.  The 
spiral  course  sequence  was  developed  with  the  aid  of  a computer 
program,  developed  by  the  investigator,  which  is  based  on  a 
methodology  known  as  Interpretive  Structural  Modeling  (ISM). 

The  study  attempted  to  compare  the  revised  and  traditional 
courses  in  terms  of  their  resulting  academic  attrition  rates  and 
perceptions  of  course  graduates  concerning  the  fidelity  of 
training  received  in  each  course  to  real-world  recruiting 
practices.  The  revised  course  was  further  evaluated  on  its 
ability  to  reduce  within-course  failures,  and  on  the  level  of 
instructor  acceptance  of  the  computer-designed  course  sequence. 
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Findings  revealed  that,  first,  absolute  academic  attrition 
rates  in  the  revised  course  were  larger  than  those  for  the 
traditional  course.  However,  the  revised  course  demonstrated  a 
steadily  declining  trend  in  academic  attritions  over  the  duration 
of  the  evaluation.  Second,  graduates'  perceptions  of  the  fidelity 
of  training  received  in  the  revised  course  showed  less  variability 
in  ratings  of  training  adequacy  than  did  those  from  graduates  of 
the  traditional  course.  Third,  the  revised  course  demonstrated  a 
dramatic  reduction  in  the  percentage  of  within-course  failures 
over  the  evaluation  period.  Finally,  instructor  acceptance  of  the 
computer-designed  course  sequence  demonstrated  a marked  increase 
across  several  evaluation  cycles. 

The  findings  of  the  study  led  the  investigator  to  conclude 
that  the  revised  course  sequence  was  at  least  as  effective  as  the 
traditional  course  sequence  in  reducing  academic  attrition  rates 
and  within-course  failure  rates.  Furthermore,  the  revised  course 


was  concluded 

to  be 

superior  to  the 

traditional 

course  in  its 

fidelity 

to 

real-world  recruiting 

practices 

according  to 

graduates ' 

perception 

ratings . 

x 


CHAPTER  I 


INTRODUCTION 


The  United  States  Navy  Recruiting  Command's  training 
organization  has  expressed  concern  over  three  aspects  of  its 
Enlisted  Recruiter  Canvasser  course:  academic  attrition, 
within-course  failures  and  course  graduates'  perceptions  of  the 
applicability  of  school  training  to  real-world  recruiting 
practices.  In  addressing  these  concerns,  the  organization's 
administration  suggested  that  the  course  be  revised  in  an  attempt 
to  improve  the  state  of  the  course  in  terms  of  these  three 
aspects.  In  a preliminary  review  of  the  literature  on  curriculum 
revision,  the  investigator  discovered  that  sequencing  of 
instruction  is  among  the  most  critical  segments  in  the  instruc- 
tional design  process.  Bloom  (1976),  for  example,  substantiated 
its  importance  in  a separate  review  of  research,  wherein  he 
suggested  that  instructional  sequencing  accounts  for  fifty  percent 
of  the  regressed  variance  in  school  achievement.  The  concept  of 
sequence  referred  to  in  Bloom's  review  is  similar  to  that  referred 
to  by  Gagne  (1962,  1968,  1977)  in  a classic  series  of  investiga- 
tions through  which  he  developed  the  notion  of  the  learning 
hierarchy . 

An  important  aspect  of  instructional  sequencing  addresses  the 
overall  organization  of  topical  elements  within  a total 
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curriculum.  Here  we  may  differentiate  between  instructional 
sequencing  on  a macrocosmic  scale  (i.e.  inter-topical),  and 
sequencing  on  a microcosmic  scale  (i.e.  intra-topical) . Gagne's 
methodology  falls  into  the  latter  category.  Indeed,  Briggs  and 
Wager  (1981),  considered  Gagne's  intellectual  skills  hierarchies 
as  components  of  more  global  instructional  curriculum  maps  which 
"can  contain  intellectual  skills  objectives  and  their  principle 
prerequisites  as  well  as  objectives  from  other  domains  . 

(p.  91) 

On  a macrocosmic  scale,  curriculum  sequences  are  classifiable 
into  two  generic  groups:  block-sequenced  and  spiral-sequenced. 
A block-sequenced  curriculum  may  be  defined  as  one  in  which 
topics  are  presented  separately  as  intact  entities.  For  example, 
topic  A is  presented  in  toto,  then  topic  B is  presented,  etc.  No 
attempt  is  made  to  relate  concepts  of  topic  A to  those  of  topic 
B.  Such  a sequencing  strategy  would  be  used  when  a curriculum  is 
composed  of  highly  unrelated  topics.  The  investigator  knows  of  no 
examples  of  curricula  which  are  purely  block-sequenced. 

A spiral-sequenced  curriculum,  described  by  Bruner  (1960),  is 
described  as  one  in  which  fundamental  concepts  are  presented  early 
in  the  curriculum,  and  are  built  upon  and  related  to  more  advanced 
concepts  as  the  curriculum  progresses.  Apparently,  curricula 
containing  highly  interrelated  topical  content  would  be  amenable 
to  a pure  spiral-sequenced  strategy. 

The  investigator  discovered  that,  in  practice,  few  if  any 
contemporary  curricula  exhibit  purely  one  or  the  other  generic 
sequencing  strategy.  Most  curricula  are  sufficiently  complex  to 
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contain  elements  of  both  strategies.  The  U.S.  Navy  Recruiting 
Command's  training  course  for  Enlisted  Recruiters  is  an  example  of 
a complex  curriculum  amenable  to  a combination  of  both  sequencing 
strategies.  Yet,  in  the  past,  the  amount  of  sprial  sequencing 
used  in  the  course  was  minimal.  Based  on  Bloom's  (1976)  review  of 
research  regarding  the  importance  of  sequence  in  instruction,  and 
Bruner's  (1960)  writings  on  spiraling,  the  investigator  proposed  a 
revision  to  the  course  in  which  a learning  hierarchy  would  be 
developed  for  the  course,  and  from  that,  a more  heavily  spiraled 
course  sequence  would  be  generated  and  implemented. 

Problem  Statement 

Would  the  implementation  of  a primarily  spiral-sequenced 
structure  for  the  course  result  in  lower  student  attrition  and 
roll-back  rates  and  a greater  perception,  in  the  minds  of  the 
course  graduates,  of  the  applicability  of  school  training  to 
real-world  recruiting  practices  than  that  observed  in  a 
predominantly  block- sequenced  course  structure? 

Objective 

The  purpose  of  this  study  was  to  ascertain  the  effectiveness 
of  a computer-generated,  spiraled  curriculum  sequence  for  a course 
of  instruction  in  Navy  Enlisted  Recruiting  practices  (the  revised 
course)  against  a previous  version  of  the  same  course  sequenced  in 
a linear,  or  block,  manner  (the  traditional  course). 
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Study  Questions 

1.  Is  there  a difference  in  academic  attrition  rates 
between  the  traditional  and  revised  courses? 

2.  Is  there  evidence  of  either  stability  or  consistent 
decline  in  academic  attrition  rates  across  several 
independent  observations  of  the  revised  course? 

3.  How  do  graduates'  perceptions  of  the  applicability  of 
training  to  real-world  recruiting  requirements  differ  in 
the  traditional  course  compared  to  those  in  the  revised 
course? 

4.  Is  there  evidence  of  a change  in  the  level  of  procedural 
fidelity  regarding  classroom  instruction  in  the  revised 
course  over  the  duration  of  the  study? 

5.  Is  there  a difference  in  academic  roll-back  rates  across 
several  independent  observations  of  the  revised  course? 

Justification  For  the  Study  Questions 

The  three  areas  of  concern  expressed  by  the  school's 
administration  were:  1)  the  rate  of  academic  attrition  from  the 

course,  2)  the  rate  of  within-course  academic  failures  and  3) 
course  graduates'  perceptions  regarding  the  applicability  of 
training  to  real-world  recruiting  practices.  The  first  study 

question  was  directly  related  to  the  first  area  of  concern. 

Nearly  every  instructional  innovation  may  be  influenced  to 
some  degree  by  either  the  Hawthorne  Effect,  wherein  the 
experimental  group's  results  are  inflated  due  to  knowledge  that 
they  are  part  of  the  experiment,  or  the  John  Henry  Effect,  where 
a control  group  produces  inflated  results  out  of  a psychological 
need  to  prove  itself  better  than  the  experimental  group  (cf. 

Anderson,  et  al.;  1973).  Conscious  of  this  potential  effect,  the 

investigator  developed  a research  design  which  incorporated  three 
independent  observations  of  the  revised  curriculum  over  a time 


5 


span  of  one  and  one-half  years.  The  second  study  question  was 
formulated  to  identify  trends  in  the  academic  attrition  rate  in 
the  revised  course  throughout  this  time  span.  Either  a stable  or 
decreasing  trend  in  the  revised  course's  attrition  rate  was 
acceptable  as  evidence  of  improvement  brought  about  by  the  revised 
course  sequence. 

Regarding  the  third  study  question,  ultimate  acceptance  of 
the  revised  course  by  the  school's  administration  was  based  large- 
ly on  the  course's  ability  to  promote  positive  transfer  of  train- 
ing from  the  school  environment  to  the  real  world  of  recruiting. 
One  measure  of  whether  such  transfer  actually  took  place  was 
assumed  to  rest  with  the  perceptions  of  the  course  graduates. 

In  attempting  to  measure  differences  with  respect  to  two 
sequences  of  instructional  presentation,  it  is  necessary  to  insure 
that  the  intended  instructional  sequence  is  actually  being 
followed  at  the  classroom  level.  Reluctance  of  the  instructional 
staff  to  follow  the  revised  sequence  may  result  in  little  more 
than  perpetuation  of  the  traditional  sequence.  If  such  an  event 
were  to  occur,  an  identified  difference  in  any  of  the  curriculum 
aspects  studied  could  not  necessarily  be  attributed  to  the  revised 
sequence.  It,  therefore,  became  necessary  to  monitor  the  instruc- 
tional process  in  terms  of  presentational  sequence.  The  fourth 
study  question  provided  for  the  gathering  of  data  regarding  how 
well  the  revised  sequence  was  followed  by  the  instructional  staff. 

The  fifth  study  question,  dealing  with  within-course  academic 
failures  (roll  backs),  represented  another  of  the  aspects  of 


interest  to  the  school's  administration. 


Since  the  Enlisted 
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Recruiter  Course  is  competency-based  and  criterion  referenced,  it 
can  be  argued  that  academic  achievement  scores  from  course 
examinations  would  show  little  if  any  change  between  the 
traditional  and  revised  courses.  The  argument  follows  from  the 
fact  that  in  this  competency-based  course,  students  must 
demonstrate  mastery  of  at  least  75%  of  the  course  objectives  to  be 
considered  competent  in  the  theory  of  recruiting.  Since  this 
competency-based  philosophy  remained  essentially  the  same  in  both 
the  traditional  and  the  revised  courses,  it  was  reasonable  to 
expect  that  mean  achievement  scores  across  the  two  courses  would 
be  essentially  stable,  within  the  limits  of  measurement  error.  A 
more  meaningful  measure,  however,  was  the  ratio  of  students  who 
were  rolled  back  one  or  more  weeks  of  the  course,  but  did 
eventually  graduate.  The  investigator  suspected  that  such  a 
measure  would  relate  more  closely  to  the  type  of  sequence  used  in 
the  course  than  would  academic  achievement. 

Assumptions  and  Limitations 

One  of  the  assumptions  in  this  study  related  to  the 
disagreement  between  Cronbach  and  Scriven  (Cronbach,  1963;  pp. 
672-83)  concerning  the  purpose  of  educational  evaluation.  Scriven 
believes  it  appropriate  to  compare  different  educational  programs 
and  to  determine  which  one  is  best.  Cronbach,  on  the  other  hand, 
posits  that  educational  programs  are  sufficiently  diverse  in  terms 
of  their  goals,  content,  target  populations,  etc.  as  to  be 
virtually  incomparable  in  any  but  the  most  general  sense.  To 
Cronbach,  the  purpose  of  educational  evaluation  is  to  answer  the 
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question:  How  can  a particular  program  be  made  better?  The 
author  agrees  with  this  latter  position.  Comparisons  and  general- 
izations between  this  particular  course  and  some  other  which 
teaches  different  topical  material  are  beyond  the  limits  of  the 
design  controls  imposed  on  this  study. 

A second  assumption  involved  the  question  of  how  faithfully 
the  instructional  sequence  was  implemented  in  both  the  revised  and 
traditional  courses  (instructional  procedural  fidelity).  Because 
records  of  classroom  observations  did  not  exist  for  the  tradition- 
al course,  the  two  course  sequences  could  not  be  compared  on  this 
dimension.  However,  it  was  determined  from  historical  documents 
that  the  traditional  course  had  been  taught  essentially  the  same 
way  for  nearly  two  years  prior  to  the  implementation  of  the  revis- 
ed course.  The  investigator  assumed  this  length  of  time  to  be 
sufficient  for  the  instructional  staff  to  stabilize  implementation 
procedures  in  the  traditional  sequence.  Furthermore,  because  data 
on  attrition  rates,  roll-back  rates  and  graduates'  perceptions  for 
the  traditional  course,  were  gathered  during  the  last  three  months 
the  course  was  in  operation,  the  investigator  felt  secure  in 
assuming  that  the  implementation  of  the  traditional  sequence  was 
stable.  Nevertheless,  because  no  evidence  existed  to  support  the 
validity  of  this  assumption,  the  study  was  limited  to  the  extent 
that  no  comparison  between  the  traditional  and  revised  course 
sequences  could  be  investigated  on  this  dimension. 

A third  assumption  involved  the  validity  of  data  gathered 
from  student  surveys  and  administrative  records.  It  was  assumed 
that  data  provided  from  these  sources  were  unbiased  and  truthful. 
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Definition  of  Terms 

Academic  Attrition  Rate  - The  ratio  of  the  number  of  subjects 
disenrolled  from  the  course  for  academic  reasons  to  the  total 
number  of  subjects  initially  enrolled  in  each  evaluation  cycle. 

Academic  Roll-back  Rate  - The  ratio  of  the  number  of  subjects 
who  failed  a week  of  instruction  for  academic  reasons  and 
were  required  to  repeat  the  week  to  the  total  number  of  subjects 
initially  enrolled  in  each  evaluation  cycle. 

Graduates'  Perceptions  of  Course  Adequacy  - A subjective 
rating  elicited  from  course  graduates  after  they  had  had  time  to 
acquire  some  experience  in  the  field.  Their  perceptions  were 
derived  from  responses  to  a post-graduate  survey  composed  of 
questions  relating  to  topics  presented  in  the  course. 

Instructional  Procedural  Fidelity  - The  degree  of  similarity 
between  the  intended  course  sequence  as  published  in  course 
schedules  and  the  actual  sequence  implemented  in  the  classroom  as 
determined  through  unobtrusive  measures. 

Revised  Course  - An  instructional  sequence  for  the 
presentation  of  recruiting  concepts  (e.g.  sales  techniques, 
prospecting  methods,  etc.)  based  upon  expert  judgement  concerning 
the  interrelationships  among  the  concepts  of  the  curriculum 
(inputs).  The  sequence  was  derived  with  the  aid  of  a computer 
program  which  manipulated  the  inputs,  made  logical  inferences 
based  on  the  interrelationships  provided  and  developed  a 


hierarchical  structure  for  the  curriculum  from  those  inferences. 
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Traditional  Course  - An  instructional  sequence  developed  for 
the  presentation  of  recruiting  concepts  (e.g.  sales  techniques, 
prospecting  methods,  etc.)  based  upon  expert  judgement  concerning 
the  interrelationships  among  the  concepts  of  the  curriculum. 
Operationally,  this  type  of  sequence  is  defined  as  one  constructed 
manually  according  to  the  procedures  described  by  Gagne  (1977)  or 
Briggs  and  Wager  (1981). 

Overview  of  the  Dissertation 

Chapter  II  contains  an  in-depth  review  of  two  topics.  First, 
the  historical  development  of  instructional  hierarchies  is 
examined.  Second,  a review  is  presented  of  various  attempts  to 
apply  rigorous  methodologies  to  the  design  of  curriculum 
sequences.  Chapter  III  describes  the  design  of  the  study.  Chapter 
IV  is  devoted  to  an  analysis  of  the  evaluation  data.  Chapter  V 
presents  a summary  of  findings,  conclusions  and  recommendations. 


CHAPTER  II 


REVIEW  OF  THE  LITERATURE 


The  Historical  Development  of  Instructional  Hierarchies 
One  distinction  between  the  curriculum  focus  in  this  country 
prior  to,  during  and  following,  the  decade  of  the  1960s  was  in 
scope.  During  the  time  in  which  Tyler  (1949)  proposed  his  theory 
of  curriculum  design,  the  emphasis  in  curriculum  was  macroscopic, 
i.e.  subj ect -centered . The  primary  question  of  interest  at  the 
time  was:  "What  subject  matter  best  suits  the  needs  of  the 

learners  for  the  challenges  they  face  today  and  in  the  future?" 
This  focus  was  essentially  the  same  as  that  of  the  early  part  of 
the  century  when  Bobbitt  (1924)  and  Charters  (1923)  first  espoused 
it.  In  fact,  the  Eight  Year  Study,  in  which  Tyler  was  key  member, 
was  predicated  on  the  need  for  learners  to  be  prepared  for 
college,  rather  than  vocational  work. 

An  Atomistic  View  of  Knowledge  - The  Spiraling  Paradigm 

By  the  beginning  of  the  1960 's,  it  became  apparent  to 

researchers  that  a macroscopic  view  of  learning  could  not  provide 
answers  to  questions  such  as,  "why  do  some  learners  learn  more 

quickly  than  others?",  and  "is  there  a way  to  make  learning  more 

effective  for  all  learners?"  Educational  psychologists,  as  well 
as  curriculum  makers,  were  looking  for  a means  of  perceiving 
instruction  in  a more  atomistic  fashion.  The  early  work  of  Jerome 
Bruner  and  others  provided  a step  in  that  direction.  In  his 
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synopsis  of  the  proceedings  of  the  Woodshole  Conference  entitled 
The  Process  of  Education,  Bruner  (1960)  detailed  a curriculum 
design  concept  known  as  "spiraling”.  The  concept  is  based  on  the 
assumption  that  to  learn  effectively,  a learner  needs  an  initial 
mental  framework  in  which  to  store  basic  concepts  and  facts  which 
would  later  be  internalized  to  form  more  complex  concepts,  rules 
and  generalizations.  Bruner  clearly  saw  a spiral  curriculum  as 
possessing  the  ability  to  effectively  teach  complex  concepts  in  an 
"intellectually  honest"  way  to  any  individual. 

Using  the  spiraling  concept,  one  would  present  an  underlying 
conceptual  base  first,  and  then  rely  on  this  conceptual  base  as  a 
foundation  upon  which  to  build  more  elaborate  combinations  of 
facts  and  related  concepts.  One  could,  for  example,  teach  a 
four-year  old  child  the  theory  of  thermodynamics.  One  simply 
calls  it  something  other  than  thermodynamics.  Indeed,  it  is  the 
rare  four-year  old  who  does  not  intimately  understand,  from 
previous  personal  experience,  the  concepts  of  heat  and  cold,  which 
are  the  fundamental  concepts  of  thermodynamics.  This  is  the  sort 
of  conceptual  base  Bruner  referred  to  as  necessary  for  learning 
more  advanced  principles. 

However,  the  paradigm  was  shown  to  have  limits.  Results  of 
attempts  to  apply  Bruner's  concept  to  actual  instruction  were 
mixed,  although  not  experimentally  verified.  Apparently,  some 
academic  subjects  were  more  amenable  to  spiral  sequencing  than 
others.  So  strong  was  the  controversy  surrounding  the  efficacy  of 
the  technique  that  Bruner  later  wrote  that  the  major  flaw  in  the 
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concept  was  that  it  made  tacit  assumptions  that  students  were 
pre-raotivated  to  learn,  and  that  all  students  had  the  benefit  of 
existing  analytic  skills  supposedly  taught  via  some  "middle-class 
hidden  curriculum”  (Bruner,  1971;  p.  19). 

Yet,  the  apparent  limitations  of  the  spiraling  paradigm  are 
reconcilable  if  considered  in  light  of  a holistic  view  of 
education.  Bruner,  for  instance,  believed  that  the  structure  of 
knowledge  was  discipline-dependent.  For  example,  he  viewed  the 
structure  of  mathematics  or  the  sciences  as  different  from  the 
structure  of  the  humanities  or  the  arts.  In  fact,  Gagne  (1962) 
believed  that  the  fundamental  difference  between  these  particular 
disciplines  was  in  the  human  faculties,  or  domains,  affected  most 
in  learning  them.  Specifically,  Gagne  saw  the  structure  of 
mathematics  as  composed  of  well-defined  concepts,  rules  and 
principles  which  could  be  discretely  quantified  through  the  use  of 
the  intellect,  whereas  the  structure  of  the  arts  was  more  closely 
linked  to  the  affect  and  less  to  the  intellect.  Thus,  Bruner 
delineated  the  boundaries  of  effectiveness  of  his  spiraling 
concept  to  include  those  tasks  and  disciplines  characterized 
primarily  as  intellectual.  As  will  be  seen,  this  delineation  was 
a generalized  formulation  of  a more  specific  notion  espoused  years 
earlier  by  Gagne. 

Hierarchically  Mapping  the  Structure  of  Knowledge 

The  concept  of  the  learning  hierarchy  resulted  from  the 
research  of  Gagne  (1962,  1968,  1977)  and  Gagne  and  Paradise 
(1961).  In  a classic  paper,  Gagne  (1968)  defined  a learning 
hierarchy  as  " . . .a  set  of  specified  intellectual  capabilities 
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having  ...  an  ordered  relationship  to  each  other"  (p.  2).  Re 
described  the  hierarchy  as  a construct  which  "represents  the  most 
probable  expectation  of  greatest  positive  transfer  [of  knowledge]" 
(p.  3).  This  is  exactly  what  Bruner  (1960)  believed  his  spiraling 
concept  would  elicit  - a path  providing  the  greatest  positive 
transfer  of  knowledge.  Like  Bruner,  Gagne  (1968)  "incautiously 
attempted  to  generalize  the  [concept]  of  learning  hierarchies  to 
such  subject  matter  fields  as  the  social  sciences"  (p.  4).  He 
realized,  however,  that  "in  doing  this,  it  is  quite  easy  to  fall 
into  the  trap  of  describing  'knowledge'  entities  rather  than  skill 
entities"  (p.  4).  This  is  the  major  departure  in  the  thinking  of 
Gagne  and  Bruner.  Whereas  Bruner  (1960)  defined  the  limits  of  his 
spiraling  concept  in  holistic  terms  (e.g.  intellectual  versus 
affective),  Gagne  (1968)  defined  the  limit  of  his  learning 
hierarchy  concept  as  one  existing  between  knowledge  and  skills. 
This  shift  in  the  view  of  the  structure  of  knowledge  clearly 
defined  the  limits  of  use  of  such  structuring  concepts  as  the 
learning  hierarchy. 

According  to  Gagne  (1968),  his  hierarchy  concept,  as  applied 
to  instructional  presentation,  is  appropriate  to  those  bodies  of 
knowledge  which  inherently  involve  acquisition  and  application  of 
learned  skills.  His  favorite  exemplar  is  mathematics,  which 
involved  both  intellectual  knowledge  and  skills.  The  difference, 
though,  between  this  skill-based  knowledge  and  the  knowledge  of 
the  social  sciences,  according  to  Gagne,  is  that  in  mathematics 
not  only  is  the  skill  dependent  upon  the  acquisition  of  the 


14 


knowledge,  but  the  knowledge  is  discretely  quantifiable.  For 

example,  knowledge  of  the  processes  of  addition  and  subtraction 
is  necessary  to  master  these  skills,  as  well  as  to  understand  the 
more  complex  processes  of  multiplication  and  division  and  the 
skills  necessary  to  master  them.  Mathematics  abounds  with  such 
interrelationships.  There  are  fewer  instances  of  similar 

interrelationships  in  the  social  sciences.  This  does  not 

denigrate  the  knowledge  of  the  social  sciences,  but  merely 

distinguishes  it  from  the  skill-based  knowledge  of  mathematics. 

The  previous  example  illustrates  the  complexity  of  the 
relationships  among  the  elements  of  mathematics.  Add  to  this 

complexity  the  need  to  consider  the  interrelationships  between  and 
among  intellectual  skills  and  verbal  skills  necessary  for  the 
student  to  explain  (as  opposed  to  perform)  a mathematical  skill, 
and  it  becomes  apparent  that  the  process  of  developing  a learning 
hierarchy  to  represent  a single  skill,  or  an  entire  discipline  of 
skills,  is  not  simple. 

The  process  proposed  by  Gagne  (1968)  for  developing  learning 


hierarchies 

consis  ted 

essentially  of 

two 

stages 

. First,  one 

identified 

the  skill 

to  be  learned, 

and 

then 

determined  its 

constituent  components  (or  tasks)  - both  intellectual  and  verbal. 
Second,  one  attempted  to  determine  precisely  how  the  various  tasks 
related  to  each  other  to  make  up  the  resultant  skill.  Gagne's 
method  for  determining  task  interrelationships  was  to  begin  with 
the  final  (terminal)  task  and  ask  the  following  question  of  it  and 
each  subsequent  task  in  the  list:  "What  would  the  individual 
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already  have  to  know  how  to  do  in  order  to  learn  this  new 
capability  simply  by  being  given  verbal  instructions?"  (p . 3). 

Once  a hierarchy  was  constructed  in  this  fashion,  it  was  analyzed 
to  determine  if  all  interrelationships  in  the  hierarchy  were 
valid,  or  if  errors  existed.  If  inconsistencies  were  shown  to 
exist  in  the  hierarchy,  the  entire  process  was  repeated  in  an 
iterative  trial-and-error  manner  until  a hierarchy  was  generated 
which  displayed  as  few  internal  inconsistencies  as  possible.  The 
technique  tended  to  become  time-consuming  as  the  number  of  tasks 
to  be  ordered,  hence  the  number  of  interrelationships  between 
tasks,  increased.  Nevertheless,  as  a result  of  his  research, 
Gagne  was  able  to  formulate  an  empirical  sequence  for  the  learning 
of  intellectual  skills  (see  Briggs  and  Wager,  1981;  pp.  90-91). 
This  sequence  combined  such  subskills  as  discrimination,  rule 
using,  concept  formation  and  problem  solving  in  a structure 
similar  to  that  shown  in  Figure  1.  An  illustration  of  Gagne's 
hierarchy  development  technique  is  provided  in  Appendix  A. 
Experimental  Validation  of  Learning  Hierarchies 

Through  the  use  of  this  questioning  process,  Gagne  was  able 
to  hierarchically  organize  several  intellectual  skills  in 
mathematics,  including  the  basic  arithmetic  operations,  grouping 
(or  set  theory),  formula  generation,  etc.  (Gagne,  1962;  1968). 

Furthermore,  he  validated  his  hierarchies  through  testing.  His 
basic  experimental  approach  was  essentially  the  same  in  each 
experiment.  First,  a test  was  administered  to  the  learner  to 
identify  which  subordinate  tasks  he  or  she  already  knew  how  to  do. 
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Figure  1.  Gagne's  Generalized  Hierarchical  Sequence  for  the 
Development  of  an  Intellectual  Skill 
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Next,  the  learner  was  taught  to  do  those  subordinate  tasks 
which  he  or  she  could  not  initially  perform.  Then,  after 
instruction,  the  learner  was  given  verbal  directions  about  doing 
the  final  (or  terminal)  task  without  any  practice  on  it. 

The  results  of  Gagne's  experiments  indicated  that  students 
who  learned  all  the  subordinate  tasks  generally  could  also  perform 
the  terminal  task  without  prior  practice.  On  the  other  hand, 
those  who  lacked  the  ability  to  perform  the  subordinate  tasks 
usually  could  not  perform  the  terminal  task  either.  Experiments 
such  as  these  formed  the  basis  for  a belief  in  a true  hierarchical 
ordering  of  subordinate  tasks  which  built  toward  a final  skill. 

In  the  early  days  of  Gagne's  theory  testing,  few  extant 
measurement  techniques  were  capable  of  measuring  the  types  of 
hierarchical  ordering  he  was  seeking.  A technique  used  by  Gagne 
was  known  as  Guttman  Scaling;  this  technique  is  based  upon  the 
premise  that  given  a series  of  tests  relating  to  a set  of 
hierarchically  ordered  topics,  say  A,  B,  C and  D as  in  Figure  2,  a 
learner  should  not  be  able  to  pass  test  D until  and  unless  he  has 
passed  all  tests  subordinate  to  it.  Thus,  a perfect  Guttman  score 
on  the  entire  series  of  tests  could  be  illustrated  in  matrix  form 
as  in  Figure  3.  In  this  matrix,  a "1"  indicates  that  passing 
the  test  identified  by  the  row  index  is  prerequisite  to  passing 
the  test  identified  by  the  column  index. 

The  application  of  Guttman  scaling  techniques  to  the  Gagne 
experiments  is  straight-forward  since  the  skill  hierarchy  is 
determined  a priori.  The  validity  of  a Gagne-type  hierarchy  was 


Figure  2.  A Hierarchically  Ordered  Set  of  Tests 
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Figure  3.  A Perfect  Guttman  Matrix 


the  measure  of  how  closely  each  learner's  test  score  sequence 
approached  the  ideal  indicated  in  Figure  3.  Because  Gagne's  skill 
hierarchies  were  usually  of  a more  complex  nature  than  the  simple 
linear  relationship  depicted  in  Figure  2,  Gagne  concluded  that 
this  scaling  technique  "did  not  seem  entirely  satisfactory  . . 
(Gagne,  1968;  p.  6).  Part  of  the  reason  for  this,  alluded  to  by 
Bergan  (personal  communication),  appears  to  derive  from  a limita- 
tion of  scalogram  analysis  when  applied  to  hierarchical  ordering. 
This  limitation  can  be  better  understood  by  the  following 
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Illustration.  Given  two  tests,  one  of  which  is  truly  prerequisite 
to  the  other,  scalogram  analysis  will  correctly  deduce  the  order. 
However,  assume  that  two  tests  (or  concepts)  are  equivalent,  such 
that  one  must  be  mastered  prior  to  acquisition  of  the  other,  and 
that  this  prerequisite  mastery  exists  in  both  directions  between 
the  two  tests.  In  other  words,  the  two  tests  are  mutually 
supportive.  According  to  Bergan,  since  scalogram  analysis  is 
sensitive  to  linear  supporting  relationships,  the  method  becomes 
confounded  in  such  an  instance,  and  attempts  to  produce  a linear 
hierarchical  ordering.  Despite  this  apparent  limitation,  Gagne 
discovered  through  his  validation  experiments  that  a significantly 
larger  percentage  of  learners  learned  according  to  his 
hierarchically  ordered  path  than  those  who  did  not. 

Further  Validation  of  Learning  Hierarchies 

After  Gagne  introduced  the  concept  of  the  learning  hierarchy 
and  demonstrated  its  utility  in  the  learning  process,  independent 
research  ensued  conducted  principally  by  the  Learning  Research  and 
Development  Center  at  the  University  of  Pittsburgh.  Wang  (1973), 
for  example,  attempted  to  validate  hierarchical  instructional 
sequences  for  an  early  learning  curriculum.  The  curriculum 
consisted  of  fourteen  hierarchically  ordered  major  instructional 
units,  while  each  unit  was,  itself,  a hierarchically  arranged 
fine-structure  of  instructional  objectives  (or  tasks).  In  one  of 
these  experiments,  Wang  used  a variation  of  Guttman  scaling  known 
as  Multiple  Scalogram  Analysis  (Lingoes,  1963)  which  essentially 
reorders  the  tests  based  upon  resulting  student  pass/fail  ratios 


20 


to  produce  the  most  optimal  ordering  sequence  possible.  For  her 
experiment,  Wang  used  a hypothesized  sequence  developed  in  an 
earlier  project  (Wang,  et  al.;  1968).  She  assumed  that  the 
hypothesized  sequence  would  be  validated  when  it  matched  the 
empirical  scale  produced  by  multiple  scalogram  analysis,  and  would 
be  rejected  if  it  differed  from  the  empirical  scale.  The 
findings  of  the  study  were  mixed.  The  hypothesized  sequence  of 
the  fourteen  major  curriculum  units  was  directly  confirmed. 
However,  the  fine-structure  sequences  of  each  of  the  units  yielded 
results  which  were  difficult  to  interpret.  Based  upon  the  results 
of  this  and  other  related  studies  (Resnick  and  Wang,  1969;  Wang  et 
al.;  1971),  Resnick  (1972)  suggested  that  there  may  be  an  optimal 
"unit  size"  for  hierarchical  analysis. 

The  work  of  Bergan,  Karp  and  Neumann  (1979)  expanded  upon 
Wang's  (1973)  research  in  an  attempt  to  determine  if  individual 
characteristics,  coupled  with  mastery  of  hierarchically 
subordinate  skills,  influenced  the  acquisition  of  super-ordinate 
skills.  They  used  a multiple  regression  procedure  to  determine 
correlations  among  various  learner  characteristics  and  subsequent 
learner  performance  on  a set  of  hierarchically  subordinate 
skills.  These  correlations  were  then  ranked  to  form  a hierarchi- 
cally ordered  arrangement  of  individual  characteristics  and  skills 
making  up  the  task  to  be  learned.  Their  findings  provided  tenta- 
tive support  for  the  contention  that  "individual  characteristics 
affect  initial  prerequisite-skill  performance  directly,  and  super- 
ordinate-skill  learning  indirectly  through  their  impact  on 
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prerequisite  skill  learning"  (Bergan  et  al. , 1979;  p.  252).  These 
findings  provided  at  least  a tentative  explanation  for  variations 
in  learner  performance. 

Thus,  according  to  Bergan,  one  explanation  for  the  less  than 
overwhelming  success  experienced  by  Gagne,  Wang  and  other 
researchers  who  attempted  to  validate  hierarchically  ordered 
instructional  sequences  "could  be  attributed  to  other  learned 
skills  that  are  prerequisite  to  the  skills  under  investigation, 
but  that  have  not  been  explicitly  identified  and  assessed"  (Bergan 
et  al.,  1979;  p.  253).  Once  again,  there  is  evidence  that  the 
process  of  developing  and  testing  learning  hierarchies  and 
instructional  sequences  is  complicated  by  interacting  factors. 

Although,  as  indicated,  there  has  been  research  in  the  area 
of  hierarchical  sequencing  of  instruction,  it  is  scant  and  limited 
in  scope.  One  reason  for  this  paucity  may  be  the  difficulty  of 
hypothesizing  and  constructing  learning  hierarchies.  Glaser 
(1972)  stated  the  problem  quite  succinctly  when,  in  his  review  of 
experiments  with  hierarchical  learning,  he  commented  that  he  had 
"not  seen  suggested  a set  of  rules  by  which  a hierarchical 
analysis  is  rigorously  and  systematically  performed  other  than 
perhaps  two  rules:  (a)  lay  out  a chain  [of  skills],  and  (b)  ask 
the  question  originally  asked  by  Gagne  . . ."  (p.  353). 

Approaches  to  the  Development  of  Instructional  Hierarchies 

Up  until  the  time  of  the  late  1970s,  much  of  the  methodology 
of  instructional  design  had  been  fairly  well  proceduralized 
(cf.  Gagne  and  Briggs,  1974),  with  the  single  exception  of 
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instructional  hierarchy  development.  In  fact,  the  extent  to  which 
formal  procedure  entered  into  this  phase  of  the  development 
process  was  described  by  such  phrases  as  "common  sense  logical 
ordering"  and  development  of  a sequence  "that  an  experienced 
designer  would  employ  in  practice"  (Gagne  and  Briggs,  1974;  pp. 
139-140).  The  need  for  formal  procedures  useable  by  novice 
curriculum  designers  was  evident. 

Davies  (1973,  pp.  94-97)  refined  a formalized  matrix  analysis 
procedure  for  identifying  speicfic  structures  involved  in  learning 
a skill.  With  his  technique,  a curriculum  designer  or  teacher 
could  analyze  a skill  to  be  learned  to  determine  if  it  involved 
the  learning  of  signals,  chains,  multiple  discriminations, 
concepts  or  principles.  The  method  required  the  construction  of 
one  or  more  matrices  describing  the  interrelationships  among 
various  tasks  making  up  the  skill  to  be  learned.  The  technique 
was  time-consuming.  Yet,  it  did  enable  the  construction  of 
teaching  sequences  via  a more  proceduralized  methodology  than  had 
existed  previously. 

A Structural  Approach  to  Hierarchy  Development 

A different  approach  to  the  problem  of  developing  a 
procedural  methodology  for  deriving  learning  hierarchies  and 
instructional  sequences,  known  as  the  Structural  Strategies  Model 
(Reigeluth  et  al.,  1978),  emerged  in  the  late  1970s.  This  model 
consisted  of  four  steps:  (1)  determine  the  type  of  structure 

which  best  represents  the  subject  matter  to  be  presented,  (2) 
develop  the  structure  by  identifying  an  elementary  set  of 
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principles,  or  a model  upon  which  all  other  principles  elaborate, 
(3)  determine  the  supporting  and  prerequisite  substructures  of  the 
tasks  to  be  learned,  and  (4)  sequence  the  elements  of  the 
structure  and  substructures. 

To  accomplish  step  one  of  this  model,  a curriculum  designer 
must  identify  the  specific  type  of  subject  matter  structure. 
Three  structures  are  described  in  the  model.  The  first,  an 

orientation  structure,  is  an  all-inclusive  structure  which 

subsumes  most  of  the  subject  matter  to  be  taught.  Examples  of 

orientation  structures  are  taxonomies,  hierarchies  and  models. 
The  second  type  of  structure  is  called  a supporting  structure.  It 
provides  knowledge  that  contributes  to  the  understanding  of  an 
orientation  structure.  It  may  also  be  in  the  form  of  a taxonomy, 
hierarchy  or  model.  Such  structures  may  be  combined  in  a 

particular  instructional  design,  similar  to  the  combination  found 
in  Wang's  (1973)  study  wherein  her  "orientation  structure"  was  the 
hierarchy  of  fourteen  instructional  units  each  containing  a 

"supporting  structure"  of  several  hierarchically  ordered 
objectives . 

A learning  prerequisite  structure  is  the  third  type  described 
in  the  model,  and  is  the  most  loosely  defined.  It  is  described  as 
containing  any  relevant  information  presupposed  by  any  other 
structure.  This  structure  relates  most  closely  to  the  prerequi- 
site skills  brought  by  the  student  into  a course  (cf.  Bergan  et 
al.;  1979). 
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To  complete  the  second  and  third  steps  of  the  model,  the 
designer  must  be  familiar  with  these  three  different  structures, 
and  knowledgeable  of  the  curriculum's  subject  matter  content.  The 
fourth  and  final  step  in  the  model  involves  the  actual  sequencing 
of  the  instructional  content. 

An  analysis  of  the  procedures  embodied  in  the  model  reveals 
that  rigor  is  involved  in  the  "front-end  analysis"  of  the  subject 
matter  (steps  1,  2 and  3 above).  However,  the  procedure  detailed 
in  the  model  for  developing  the  instructional  hierarchy  and 
sequence  is  essentially  the  same  as  the  process  espoused  by  Gagne 
and  Briggs  (1974). 

Viewing  Curriculum  as  a Complex  System 

Another  methodology  for  developing  learning  hierarchies  and 

instructional  sequences  is  one  based  upon  a theory  of 

complex-system  analysis  known  as  Interpretive  Structural  Modeling 

(Warfield,  1976).  The  premise  behind  this  theory  is  that  any 

complex  system  of  interacting  elements  is  capable  of  being 

decomposed  into  a constituent  hierarchical  structure  which 

completely  describes  the  interrelationships  existing  in  the 

system.  The  theory's  application  to  education  in  general,  and 

instructional  design  in  specific,  becomes  apparent  when  one 

recognizes  the  actual  complexity  of  the  educational  process.  As 

Warfield  (1976)  observes, 

education  involves  not  only  buildings,  teachers 
and  curriculum,  but  also  home  life,  ethnic 
attitudes,  motivation  to  learn,  economic  effects 
exemplified  by  the  'taxpayer  revolt,'  effects  of 
the  power  structure,  political  implications, 
applicable  legislation,  discrimination  and 
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more.  A study  of  education  that  fails  to  give 
proper  consideration  to  these  factors  may  be  no 
more  than  partially  sucessful,  at  best.  . . . 

Failure  to  recognize  potential  interactions 
often  precludes  action  that  could  be  very 
beneficial,  and  allows  many  good  opportunities 
to  be  missed,  (p.  5) 

Of  course,  there  is  no  startling  revelation  here.  Warfield's 
observation  is,  in  fact,  generic  to  any  real-world  complex 
system.  However,  the  relevance  of  education's  complexities  to  the 
instructional  design  problem  becomes  important  when  one  realizes 
that,  up  to  this  point  in  our  review  of  rigorous  methodologies,  we 
have  not  seen  a model  which  has  the  capability  of  addressing  such 
factors  in  any  way  except  post  hoc.  Interpretive  Structural 
Modeling  (ISM)  appears  to  possess  a mechanism  capable  of  not  only 
considering,  but  also  sorting  out  the  interacting  effects  of 
limiting  constraints  while  the  complex  system  is  being  resolved 
into  its  constituent  components,  not  after  the  fact.  By  way  of 
the  methodology,  one  is  able  to  develop  hierarchical  maps 
describing  the  interrelationships  among  the  elements  of  the 
system. 

As  noted,  a curriculum  can  be  viewed  as  a complex  system  of 
interacting  elements.  Some  of  the  elements  would,  of  course,  be 
the  objectives  (or  skills)  to  be  learned.  Other  elements  would  be 
the  facilities,  budgetary,  logistic  and  staff  considerations  which 
always  interact  with  the  ultimate  curriculum  sequence,  but  which 
are  seldom,  if  ever,  considered  in  developing  instructional 
hierarchies  or  sequences;  and  not  at  all  alluded  to  in  the 
literature  reviewed  up  to  this  point.  An  application  of  the  ISM 
methodology  is  provided  in  Appendix  B. 
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Sato  (1979)  was  the  first  to  publish  an  application  of  ISM  to 
the  specific  development  of  instructional  hierarchies.  Two  years 
later,  Renckly  (1981)  reported  an  independent  effort  to  apply  ISM 
to  both  the  development  of  instructional  hierarchies  and  the 
generation  of  instructional  sequences.  Though  developed  indepen- 
dently, the  approaches  are  similar.  One  of  the  differences 
between  them  is  that  Sato's  approach  employed  a main-frame 
computer,  whereas  Renckly 's  approach  was  to  utilize  a microproces- 
sor. Another  difference  was  the  lack  of  an  explicitly  stated 
procedure  in  Sato's  work,  while  Renckly  (1981)  illustrated  an 
explicit  five-step  procedure  for  implementing  the  ISM  approach  to 
aid  in  developing  instructional  hierarchies  and  presentation 
sequences : 

1.  Identify  objectives  of  the  curriculum. 

2.  Determine  a transitive  relation  that 
applies  to  the  objectives  in  the  context 
of  the  instructional  situation. 

3.  Place  the  objective  relationships  into  a 
matrix  format  . . . 

4.  Manipulate  the  matrix  into  a suitable 
form  - called  a reachability  matrix. 

5.  Re-order  the  rows  and  columns  of  the 
reachability  matrix  and  partition  it  to 
reflect  hierarchical  levels  . . . (p.  203) 

In  following  this  process,  it  is  assumed  that  the 
instructional  objectives  are  developed  a priori.  This  activity  is 
well-developed  and  highly  proceduralized  (cf.  Gagne,  1977;  Briggs 
and  Wager,  1981).  However,  the  five-step  process  above,  though 
relatively  simple  in  appearance,  describes  a method  for  deriving 


instructional  hierarchies  which  had  been  deficient  in  the 
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literature  up  until  its  publication.  Rather  than  relying  on  a 
purely  manual  process  for  determining  an  ordered  hierarchical 
display  for  a set  of  instructional  objectives,  ISM  presents  the 
designer  with  questions  regarding  the  existence  or  non-existence 
of  relationships  between  each  pair  of  objectives  under  consider- 
ation, and  utilizes  iterative  algorithms  to  manipulate  the  infor- 
mation obtained  from  the  designers  into  a hierarchical  form.  The 
procedural  structure  offered  by  the  ISM  methodology  may,  indeed, 
possess  the  potential  to  transform  the  task  of  instructional 
design  into  a more  "scientific  art." 

In  summary,  the  tasks  of  instructional  hierarchy  building  and 
sequence  generation  have  traditionally  stressed  an  essentially 
quasi-proceduralized , trial -and -error  development  process.  Using 
such  a methodology,  researchers  have  made  significant  discoveries 
regarding  the  hierarchical  ordering  of  learning  tasks,  which  may 
be  integrated  into  some  future  theory  of  learning.  However,  it 
has  been  surmised  that  the  paucity  of  research  into  hierarchical 
learning  processes  may  be  due,  in  part,  to  the  lack  of  a 
proceduralized , easy-to-use  methodology  for  developing  such 
hierarchies . 

We  have  seen  two  rigorous  methodologies  for  constructing 
hierarchies.  Although  substantially  different  in  approach,  their 
purposes  are  similar  - to  aid  in  the  construction  of  instructional 
hierarchies  and  presentation  sequences  to  make  learning  more 
effective.  Of  these  two  methods,  Structural  Strategies  and 
Interpretive  Structural  Modeling,  the  latter  appears  to  possess 
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the  ability  to  address  instructional  constraints  via  the  same 
procedural  mechanism  used  to  address  relationships  between  and 
among  the  instructional  objectives. 


CHAPTER  III 


DESIGN 

Evaluation  Methodology 

This  study  was  a three-phase  summative  evaluation  of  the 
U.S.  Navy's  Enlisted  Recruiter  Course  - a course  designed  to  train 
sailors  the  skills  necessary  to  recruit  young  men  and  women  into 
Naval  service.  The  evaluation  methodology  used  for  this  study 
was  based  on  a strategy  developed  by  Robert  E.  Stake  (1967)  known 
as  the  Countenance  Model  (see  Figure  4).  Essentially,  the 
methodology  identifies  the  ANTECEDENTS  (those  things  which  are 
required  of  students,  staff  and  the  instructional  program  prior  to 
actuating  the  curriculum),  the  TRANSACTIONS  (the  precise 
instructional  procedures  carried  out  during  the  course  which 
affect  student  learning)  and  the  OUTCOMES  (or  products)  of  the 
course  of  instruction. 

These  three  dimensions  (Antecedents,  Transactions,  Outcomes) 
are  each  applied  to  four  general  areas  of  concern:  the  theoreti- 
cal INTENTS  of  the  course,  the  OBSERVATIONS  of  what  actually 
occurs  throughout  the  course,  pre-established  STANDARDS,  and 
JUDGEMENTS  based  upon  the  degree  to  which  those  STANDARDS  are 
met.  As  Stake  (1967,  p.  536)  described  it,  "Judging  is  assigning 
a weight,  an  importance  to  each  set  of  standards.  Rational  judge- 
ment in  educational  evaluation  is  a decision  as  to  how  much 
attention  to  pay  to  the  standards  ...  in  deciding  whether  or  not 
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DESCRIPTION  MATRIX  JUDGEMENT  MATRIX 


Figure  4.  The  Countenance  Model  of  Evaluation 
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to  take  some  administrative  action."  The  investigator  will  make 
judgements  and  recommendations  concerning  the  curriculum. 
However,  the  ultimate  decision  as  to  the  implementation  of  those 
recommendations  rests  with  the  school's  administrators. 

The  standards  considered  in  the  Judgement  Matrix  were 
absolute  standards,  i.e.  they  were  considered  solely  with  respect 
to  the  course  under  study  - in  terms  of  the  course's  own 
objectives.  The  other  type  of  standards,  known  as  relative,  would 
have  involved  comparisons  between  factors  studied  in  the  present 
course  and  similar  factors  in  other  comparable  courses  (e.g.  Army, 
Air  Force  and  Marine  Corps  Recruiting  Courses) . In  referring  to 
the  issue  of  relative  versus  absolute  standards,  Stake  (1973,  p. 
123)  has  paraphrased  Cronbach's  contention  that  "the  difference  in 
purpose  of  various  programs  being  compared  is  likely  to  be 
sufficiently  great  to  render  uninterpretable  any  outcome  other 
than  the  across-the-board  superiority  of  one  of  them."  In 
deference  to  this  contention,  the  investigator  decided  that 
evaluation  of  relative  standards  would  cloud  the  purposes  of  the 
current  study,  and  divert  attention  to  more  ancillary 
considerations.  Thus,  this  study  was  concerned  only  with 
judgements  based  on  absolute  standards  internal  to  the  revised 
course. 

The  Populations  to  be  Studied 

Three  distinct  populations  were  studied,  each  composed  of 
five  classes  of  students  in  the  Enlisted  Recruiter  course  of 
instruction  at  the  Navy  Recruiting  Orientation  Unit,  in  Orlando, 
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Florida.  The  populations  included  111,  222  and  101  subjects, 
respectively.  The  primary  reason  for  the  fluctuation  was  due  to 
the  school's  lack  of  control  over  student  loading  into  the 
course.  Three  cases  were  deleted  from  the  study,  two  in  the  first 
group  and  one  in  the  third.  These  deletions  were  due  to  circum- 
stances beyond  the  school's  control,  and  were  unrelated  to 
academics.  This  brought  the  size  of  the  three  populations  to  109, 
222  and  100  respectively. 

Because  separate  groups  of  subjects  were  studied  at  widely 
spaced  points  in  time,  they  were  considered  independent 
populations,  rather  than  samples  of  some  larger  homogeneous 
group.  Of  course,  considering  each  group  as  an  intact  population 
precluded  the  ability  to  make  statistical  inferences  from  the 
findings  of  the  study  to  a larger  group.  However,  in  terras  of  the 
evaluative  nature  of  this  study,  such  inferences  were  considered 
unnecessary. 

Instrumentation  and  Data  Collection 

To  provide  data  for  answering  the  study  questions  posed 
earlier,  several  types  of  data  were  collected  and  analyzed.  The 
questions  concerning  student  attrition  rate,  for  example,  required 
the  collection  of  specific  data  regarding  numbers  of  students  in 
each  evaluation  cycle  who  had  academically  failed  the  course.  The 
school's  personnel  department  provided  these  data  to  the 
investigator.  The  attrition  data  were  categorized  by  the 
personnel  department  according  to  reason  for  attrition.  Of  the 
reasons  tabulated,  only  those  directly  related  to  academic 
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performance  were  considered  in  this  study  (e.g.  consistent 
academic  performance  below  75%  in  weekly  exams,  and  inability  to 
perform  activities  involving  sales  skills  to  a 75%  minimum 
level).  Other  attrition  categories  (e.g.  medical  or  personal 
problems,  alcohol  and  drugs,  lack  of  a valid  driver’s  license, 
etc.)  were  not  considered  in  the  attrition  rate  computations,  as 
these  had  no  bearing  on  the  curriculum  content  or  structure  of  the 
course. 

The  question  concerning  student  roll-back  rate  required  data 
on  the  number  of  weekly  course  failures  which  resulted  in  the 
student  repeating  the  failed  week  of  instruction,  but  did  not 
result  in  immediate  attrition.  These  data  were  available  from 
students'  academic  records.  No  distinction  was  made  between  those 
students  who  were  rolled-back  once  and  those  rolled-back  more  than 
once  in  the  course.  It  was  decided  that  such  a distinction  would 
not  provide  information  useful  in  this  particular  study.  This 

decision  was  based  on  the  results  of  an  earlier  pilot  study 
conducted  by  the  investigator  which  revealed  no  consistent 
correlation  between  the  number  of  roll-back  incidents  per  student 
and  whether  or  not  the  student  eventually  failed  the  course. 
Several  students  in  each  evaluation  cycle  were  administratively 
rolled-back  for  non-academic  reasons,  such  as  medical  or  emergency 
leave,  etc.  As  was  the  case  with  attrition  data,  such  instances 
were  not  counted  in  the  final  roll-back  rate  computations. 

The  question  concerning  graduates'  perceptions  of  course 
relevance  to  real-world  recruiting  practices  was  addressed  with 
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data  gathered  via  a post-graduate  feedback  survey.  The  survey  was 
mailed  to  course  graduates  three  months  after  graduation.  The 
investigator-developed  survey  consisted  of  36  questions,  each 
based  on  a major  course  objective  (cf.  Appendixes  A and  B) . 
Graduates  were  instructed  to  rate  their  perceptions  of  the 
training  they  received  on  these  course  objectives  in  terms  of 
their  first  three  months  of  recruiting  experience.  The  rating 
scale  used  was  a five-point  Likert-type  scale: 

1.  Task  not  performed;  eliminate  from  training 

2.  Training  more  than  adequate;  reduce  emphasis 

3.  Training  adequate  for  the  task 

4.  Training  less  than  adequate;  increase  emphasis 

5.  Training  requires  MUCH  more  emphasis  in  school 

Feedback  was  solicited  from  graduates  of  each  evaluation  cycle  of 
the  revised  course,  as  well  as  from  graduates  of  the  traditional 
course. 

Data  concerning  instructional  procedural  fidelity  were 
provided  by  means  of  an  in-place  administrative  report  detailing 
usage  of  the  school's  closed-circuit  audiovisual  system.  During 
development  of  the  revised  course  sequence,  weekly  course 
schedules  containing  audiovisual  requirements  were  created  as 
operating  documents  for  instructors  and  audiovisual  technicians. 
The  technicians  were  required  to  document  all  audiovisual  usage  on 
a daily  basis  for  each  week  of  instruction.  The  documentation 
identified  three  predominant  categories  for  each  week  of 
instruction:  1)  audiovisual  sessions  which  occurred  as  scheduled, 

2)  audiovisual  sessions  which  either  did  not  occur  as  scheduled, 
or  occurred  at  other  than  scheduled  times,  and  3)  audiovisual 
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sessions  requested  by  instructors  but  not  appearing  on  the  appro- 
priate weekly  schedule.  Any  reported  incidents  falling  into 
category  #2  were  considered  course  deviations  only  if  the  actual 
showing  of  the  audiovisual  session  differed  from  that  scheduled  by 
more  than  one  hour.  During  the  first  evaluation  cycle,  this 
leeway  was  granted  because  of  instructor  uncertainty  in  estimating 
the  amount  of  classroom  time  needed  to  teach  a subject  under  the 
revised  course  sequence.  For  the  second  and  third  evaluation 
cycles,  the  leeway  was  reduced  to  30  minutes.  The  need  for  leeway 
allowances  was  deemed  necessary  by  the  investigator,  even  in  the 
later  evaluation  cycles,  because  of  anticipated  time  fluctuations 
which  normally  occur  in  even  the  most  routine  classroom  trans- 
actions, due  to  unanticiapted  student  questions,  discussions 
designed  to  clarify  student  misunderstandings,  etc. 

Instrument  Validity 

Other  than  in-place  administrative  reports  required  by  the 
school,  the  only  data  collection  instrument  used  in  the  study  was 
the  graduates'  feedback  survey  (see  Appendix  C) . The  investigator 
assumed  that  both  of  these  sources  provided  valid  data.  The 
survey  was  composed  of  items  which  identically  matched  objectives 
taught  in  the  course.  Furthermore,  the  course  objectives  were 
originally  derived  from  a job  listing  of  tasks  routinely  performed 
by  professional  enlisted  Navy  Recruiters.  During  the  early 
development  phases  of  the  course,  tasks  were  mapped  into  either  a 
one-to-one  or  a many-to-one  correspondence  with  the  course 
objectives.  The  many-to-one  correspondence  scheme  was  used  to 
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consolidate  clearly-related  task  elements.  Appendix  D contains 
the  Recruiter  Canvasser  Job  Task  Inventory,  Appendix  E contains 
the  Recruiter  Canvasser  course  objectives  derived  from  the  job 
task  inventory,  and  Appendix  F displays  the  correspondence  between 
job  tasks  and  course  objectives. 

Potential  Error  in  Measurement 

There  were  four  variables  of  interest  in  the  current  study: 
1)  academic  attrition  rate  in  each  evaluation  cycle,  2)  academic 
roll-back  rate  in  each  evaluation  cycle,  3)  graduates'  perception 
ratings  of  course  relevance  to  real-world  recruiting  practices  in 
several  content  areas,  and  4)  instructional  procedural  fidelity, 
as  measured  by  audiovisual  usage  records.  Each  of  these  variables 
was  measured  in  every  evaluation  cycle  of  the  revised  course.  In 
the  traditional  course,  only  academic  attrition  data  and 
graduates'  survey  results  were  available  from  historical  records. 

The  potential  measurement  error  for  the  first  two  variables 
dealt  with  accurately  gathering  the  data.  In  the  traditional 
course,  only  historical  records  of  academic  attrition  data  were 
available.  More  importantly,  the  school  had  no  historical  records 
to  indicate  the  criteria  which  governed  academic  attrition 
procedures  in  that  course.  The  lack  of  this  information  forced 
the  investigator  to  assume  that  the  operative  criteria  governing 
academic  attritions  in  the  traditional  course  were  the  same  as 
those  used  in  the  revised  course. 

The  third  variable,  the  perception  ratings  of  course 
graduates,  was  subject  to  a potential  sampling  error  arising  from 


37 


the  possibility  that  the  return  rate  from  the  graduates  would  be 
so  small  that  a confident  prediction  of  the  graduates'  true 
perceptions  could  not  be  made.  To  overcome  this,  a minimum  sample 
size  was  computed  to  determine  the  minimum  number  of  surveys 
needed  to  be  mailed  to  ensure  at  least  a 20%  return  rate. 
According  the  the  Navy's  Training  Analysis  and  Evaluation  Group, 
this  20%  figure  represented  an  acceptable  return  rate  for  mailed 
questionnaires  (R.  Braby,  personal  communication,  September  20, 
1980).  The  computations,  which  are  detailed  in  Appendix  G, 
yielded  a sample  size  of  100%  of  the  populations  studied.  Thus, 
surveys  were  mailed  to  all  course  graduates  from  each  evaluation 
cycle. 

The  fourth  variable,  instructional  procedural  fidelity,  was 
potentially  affected  by  errors  in  recording  deviations  from  the 
published  course  schedules.  Since  there  was  no  subjectivity 
involved  in  identifying  schedule  deviations  (all  deviations  were 
apparent  from  reading  the  published  schedule) , the  only  potential 
errors  were  those  caused  by  the  technicians  in  forgetting  to 
annotate  deviations  as  they  occurred.  The  amount  of  such  error  is 
unknown. 

Justification  for  Multiple  Evaluations 

There  was  reason  to  suspect  that  the  Hawthorne  effect  could 
potentially  inflate  any  differences  discovered  in  favor  of  the 
revised  course.  To  adjust  for  this  potential  inflation,  the 
second  and  third  rounds  of  evaluation  were  planned  to  be  conducted 
at  one,  and  one  and  one-half  year  intervals  after  the  initial 
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evaluation.  If  differences  among  the  three  evaluations  were  found 
to  either  consistently  decrease  or  stabilize  at  a positive  level 
in  favor  of  the  revised  course,  then  it  would  be  concluded  that 
the  revised  course  sequence  was  at  least  as  effective  as  the 
sequence  of  the  traditional  course  regarding  the  variables 
studied . 

Additional  Analysis  Strategies 

The  Countenance  Evaluation  Model  possesses  its  own  analysis 
strategies  designed  to  ensure  the  internal  validity  of  the  model 
as  it  is  applied  to  the  instructional  program  under  investigation. 
Specifically,  the  data  generated  in  the  Description  Matrix  (see 
Figure  1)  are  analyzed  to  determine  the  answer  to  three  principal 
questions.  The  first  question,  To  what  degree  do  the  antecedents, 
transactions  and  outcomes  come  to  pass?,  is  a measure  of  the 
congruence  between  what  is  intended  and  what  is  observed  to 
occur.  Two  important  points,  however,  need  to  be  made.  "First, 
it  may  be  that  incongruence  (little  or  no  match  between  intents 
and  observations)  is  desirable  in  the  long  run,  especially  if  the 
intents  can  be  shown  to  be  invalid  for  some  reason.  Second, 
congruence  does  not  assure  validity,  only  fidelity  ..."  (MacKay 
and  Maguire,  1971;  p.  31).  This  second  point  is  readily  apparent 
by  recognizing  that  the  concept  of  congruence  is  similar  to 
reliability,  and  further,  that  reliability  is  a necessary  but  not 
sufficient  condition  for  validity. 

The  second  question  is  of  the  following  form:  If  the 
students  and  the  instructional  program  possess  all  the  intended 
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antecedents,  and  if  the  instruction  follows  all  the  intended 
transactions,  is  it  reasonable  to  expect  the  intended  outcomes  to 
occur?  This  question  is  concerned  with  the  logical  (or  intended) 
contingency  of  the  course  being  evaluated. 

The  third  question  to  be  asked  is  similar  to  the  second. 
However,  rather  than  dealing  with  intents,  this  question  deals 
with  actual  observations.  The  question  is  of  the  following  form: 
When  the  antecedents  are  verified,  and  when  the  instructional 
transactions  actually  occur,  do  the  observed  outcomes  occur  as 
intended?  This  question  concerns  the  empirical  (or  observational) 
contingency  of  the  course. 

Descriptive  Analysis 

Analyzing  the  descriptive  data  for  congruence,  logical 
contingency  and  empirical  contingency  provides  information  useful 
in  making  recommendations  and  decisions  to  retain,  alter  or 
abandon  portions  of  the  instructional  program.  Additionally, 
these  analyses  provide  information  which  indicates  whether  or  not 
initial  intentions  were  realistic.  The  information  is  useful  for 
designing  follow-up  evaluations  of  the  same  program,  which  should 
evolve  toward  more  realistic  determinations  of  intended  antece- 
dents, transactions  and  outcomes.  Such  a process  leads  to  more 
realistic  judgements  and  stronger  recommendations  based  on  those 
judgements.  These  three  analytical  relationships  are  illustrated 
in  Figure  5.  (Note:  Numbered  paragraphs  are  used  to  facilitate 
cross-referencing  between  the  sections  which  follow.) 
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DESCRIPTIVE  DATA 


OBSERVED  DATA 


CONGRUENCE 


CONGRUENCE 


CONGRUENCE 


Figure  5.  The  Relationships  Among  Three  Types  of  Analysis  in 
the  Countenance  Evaluation  Model 
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Intended  antecedent 

1.  Entering  Students  possess  no  previous  knowledge  of  course 

material.  Data  on  this  antecedent  were  necessary  to 
determine  the  effects  on  the  other  findings  of  the  study. 

Data  sources  and  procedures  for  observing  antecedent 

1.  A course  pretest  was  used  to  measure  student  entry  level 

knowledge  of  recruiting.  The  pretest  was  administered  to 
subjects  in  the  first  evaluation  cycle.  The  test  was 
composed  of  a sample  of  questions  from  weekly  course  exams. 
It  was  expected  that  students  with  no  previous  knowledge  of 
course  material  should  score  at  an  average  guess  rate  of  25% 
or  less.  This  rate  was  determined  because  the  pretest 
contained  multiple-choice  questions  with  four  alternatives 
each. 

Intended  transactions 

1.  The  course  is  structured  into  five  week-long  modules.  These 
modules  are  sequenced  in  a spiral  fashion,  with  subsequent 
weeks'  instruction  designed  to  elaborate  on  concepts 
presented  in  the  earlier  weeks  of  the  course. 

2.  To  ensure  that  all  classes  of  students  receive  the  same 
amount  of  instruction  in  each  week,  published  course 
schedules  must  be  strictly  adhered  to  insofar  as  the  timing 
of  topic  presentations  and  audio-visual  presentations. 

3.  Course  objectives  specify  student  performance  in  behavioral 
terms  (both  cognitive  and  psychomotor) . These  objectives 
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should  be  patterned  closely  after  actual  tasks  performed  by 
recruiters  in  the  field. 

Data  sources  and  procedures  for  observing  transactions. 

1-2.  Physical  observation  of  classroom  transactions  was  required 
to  adequately  monitor  the  spiraling  of  instructional  content 
from  week  to  week,  and  to  ensure  that  course  schedules  were 
being  adhered  to  as  much  as  practical.  The  essence  of  a 
spiral  instructional  sequence  requires  that  concepts 

presented  early  in  the  course  be  frequently  referred  to 
throughout  the  course  and  integrated  into  new  concepts  taught 
later.  Observations  of  the  instructional  sequence  should  be 
accomplished  unobtrusively  with  the  aid  of  audiovisual 
records  indicating  usage  rates  of  the  closed  circuit  tele- 
vision (CCTV)  system.  By  comparing  this  record  against  the 
published  course  schedule,  protracted  sequence  breaks  can  be 
monitored  precisely.  Because  the  revised  course  employed 
CCTV  during  every  day  of  the  course,  this  method  of  sequence 
checking  was  assumed  to  be  a valid  means  of  verifying 
adherence  to  the  course  schedule. 

3.  Surveys  of  course  graduates  were  to  be  solicited  three  months 
after  graduation  to  gather  data  concerning  the  real-world 
applicability  of  course  content  as  perceived  by  the 

graduate.  Mean  perception  ratings  were  computed  for  each 
survey  item  from  the  raw  data  provided  by  respondents. 
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Intended  outcomes 

1.  Academic  attrition  rates  in  the  revised  course  were  expected 

to  show  a decreasing  trend  over  the  duration  of  the 
evaluation  study. 

2.  Academic  roll-back  rates  measured  across  all  evaluation 

cycles  were  expected  to  either  decrease  or  stabilize. 

Because  there  were  no  data  available  on  roll-back  rates  from 
the  traditional  course,  it  was  only  possible  to  analyze  roll- 
back rates  across  different  evaluation  periods  of  the  revised 
course. 

Data  sources  and  procedures  for  observing  outcomes 

1,  2.  Data  were  available  weekly  concerning:  1)  the  number  of 

academic  attritions  from  the  course,  and  roll-backs  into 
previous  weeks  of  instruction,  2)  the  class  to  which  the 
students  were  initially  assigned  and  3)  the  reason  for 
attrition  or  roll-back.  All  attritions  and  roll-backs  were 
tallied  each  week  and  accumulated  for  each  evaluation 
cycle.  Attrition  rate  (A)  and  roll-back  rate  (R)  were 
computed  as  follows: 

_ Total  academic  attritions  per  evaluation  period 

Total  population  enrolled  during  the  period 

R _ Total  academic  roll-backs  per  evaluation  period 

Total  population  enrolled  during  the  period 
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Logical  Contingency  Analysis 

The  need  for  the  Intended  Antecedent  lies  in  the  possibility 
that  students'  prior  knowledge  of  course  material  could  influence 
the  ultimate  transactions  and  outcomes  by  inflating  their  apparent 
effectiveness.  Thus,  prior-knowledge  level  must  be  assessed  and 
considered  in  any  decisions  regarding  the  merit  and  worth  of  the 
revised  instructional  program. 

The  first  two  Intended  Transactions  were  designed  to 
determine  if  the  intended  spiral  sequence  was  actually  implemented 
as  reflected  in  the  course  schedules.  The  third  Intended 
Transaction  was  designed  to  compare  the  theoretical  fidelity  of 
course  content  to  actual  field  recruiting  tasks  against  the 
fidelity  perceived  by  course  graduates. 

Both  Intended  Outcomes  relate  to  the  effectiveness  of  the 
revised  course  in  terms  of  its  ability  to  reduce  or  stabilize  the 
rates  of  academic  attrition  and  roll-back.  Analysis  of  these 
rates  across  all  three  evaluation  cycles  may  show  a trend.  It  was 
expected  that,  if  trends  occurred,  they  would  be  either  decreasing 
or  stable.  Such  trends  would  lend  support  to  the  long-term 
effectiveness  of  the  revised  spiral-sequenced  curriculum, 
regarding  attrition  and  roll-back  rates. 

The  Congruence  between  what  is  intended  and  what  is  observed, 
as  well  as  Empirical  Contingency  between  observed  antecedents, 
transactions  and  outcomes  must  await  presentation  of  the 
observational  data.  These  issues  are  discussed  in  Chapter  IV. 


CHAPTER  IV 


FINDINGS 


In  addition  to  the  Description  Matrix  presented  earlier,  the 
Countenance  Model  of  Evaluation  also  contains  a Judgement  Matrix 
made  up  of  STANDARDS  and  JUDGEMENTS  (see  Figure  4).  By  measuring 


differences 

between 

the 

Intents  and 

the  Observations  of  what 

transpired 

during 

the 

evaluation 

against  pre-established 

Standards , 

Judgements 

were 

made  as  to 

the  efficacy  of  the  revised 

curriculum.  As  mentioned,  the  Standards  used  in  this  study  were 
absolute  - that  is,  they  did  not  contain  criterial  references  to 
external  courses.  The  following  numbered  paragraphs  delineate  the 
Standards  used  in  this  evaluation  study;  they  correspond  with 
identically  numbered  paragraphs  in  the  previous  chapter. 

Standards  For  Antecedent 

1.  Class  averages  for  the  course  pretest  should  not  exceed  25%, 
the  estimated  average  guess  rate  for  a test  composed  of 
multiple  choice  questions  with  four  alternatives  each.  If  the 
mean  pretest  score  of  the  population  does  not  exceed  25%,  then 
the  indended  antecedent,  that  students  entering  into  the 
courses  possess  no  substantial  previous  knowledge  of  course 
material,  is  verified. 
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Standards  For  Transactions 

1,  2.  Course  deviations  and  unscheduled  audiovisual  requests, 
which  were  used  to  determine  the  level  of  instructional 
procedural  fidelity,  were  expected  to  exist  regardless  of 
the  type  of  course  sequence  in  effect.  This  was  expected 
because  part  of  the  school's  philosophy  was  to  encourage 
the  instructor  staff  to  experiment  with  new  instructional 
methods  and  teaching  techniques  in  the  classroom,  which 
had  the  potential  to  cause  various  degrees  of  deviation 
from  the  published  schedule.  Therefore,  overall  findings 
which  indicated  a declining  or  stabilizing  trend  in 
schedule  deviations  were  acceptable  as  evidence  that  the 
instructional  staff  had  adapted  to  and  were  willing  to 
follow  the  revised  course  sequence. 

3.  Course  objectives  should  be  patterned  after  actual  tasks 
performed  by  recruiters  in  the  field.  This  helps  to  ensure 
that  school-based  training  is  relevant  to  actual  field 
operations.  Early  in  the  course's  development,  the 

school's  administration  determined  that  some  recruiting 
tasks  were  neither  feasible  nor  economical  to  train  in  a 
resident  school  setting.  Thus,  after  its  development,  the 
Recruiter  Job  Task  Inventory  (Appendix  D)  was  subdivided 
into  two  categories  - those  tasks  which  were  trainable  at 
the  resident  school,  and  those  which  were  not.  The 
subdivision  was  performed  by  a team  of  experienced 
recruiters  and  in-house  curriculum  designers.  This 
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subdivision  is  indicated  in  Appendix  D as  follows:  tasks 

to  be  trained  at  school  are  shown  in  bold  print,  while  all 
other  tasks  are  indicated  in  normal  print.  Therefore,  if 
course  objectives  can  be  linked  to  actual  tasks  from  the 
inventory,  there  is  support  for  the  contention  that  the 
course  was  designed  to  train  to  actual  recruiting 
practices.  Furthermore,  if  analysis  of  graduates'  surveys 
indicates  that  graduates  of  the  revised  course  perceive 
their  training  as  more  adequate  for  performing  real-world 
recruiting  tasks  than  do  those  graduates  of  the  traditional 
course,  then  both  of  these  findings  can  be  taken  together 
as  evidence  that  the  revised  course  design  actually  did 
result  in  an  improved  perception  of  the  fidelity  of 
resident  training.  An  opposite  finding  would,  of  course, 
indicate  that  the  traditional  course  was  perceived  to 
possess  a greater  fidelity  to  real-world  recruiting. 

Standards  For  Outcomes 

1,2.  It  was  impractical  to  expect  either  academic  attritions  or 
roll-backs  to  be  eliminated  entirely  as  a result  of  the 
spiral  course  design.  Although  it  can  be  argued  that  a 
true  competency-based  curriculum  generates  no  failures,  in 
the  case  of  a military-related  training  course  such  as  the 
one  under  study,  there  are  realistic  constraints  of  time 
and  budget  that  necessitate  the  imposition  of  attrition 
criteria.  Also,  in  this  particular  course,  the  Recruiting 
Command  Headquarters,  in  response  to  Congressional 
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directives,  imposed  a maximum  ceiling  on  attritions  of  no 
more  than  25%  of  the  total  student  enrollment.  If  the 
course  exceeded  this  ceiling,  it  had  to  be  examined  and 
revised  as  necessary.  Thus,  within  the  framework,  of  these 
two  constraints,  findings  that  indicated  a declining  or 
stabilizing  trend  in  academic  attritions  and  roll-backs 
were  accepted  as  evidence  that  the  revised  course  was  at 
least  as  effective  as  the  traditional  course  in  these  two 
areas. 

The  results  of  each  evaluation  cycle  are  presented  in 
separate  sections  of  this  chapter.  Following  the  convention 
established  in  Chapter  III,  numbered  paragraphs  are  again 
used  to  facilitate  cross  referencing  between  and  among  the 
sections  which  follow. 

Results  of  Evaluation  Cycle  One 
Actual  Observation  of  Antecedent 

1.  To  determine  whether  or  not  students  possessed  prior 
knowledge  of  course  content,  course  pretest  scores  were 
analyzed  to  determine  if  they  differed  from  the  presumed 
minimum  25%  guess  rate.  Table  1 lists  descriptive  statistics 
for  the  pretest  scores  of  each  of  five  classes  observed 
during  evaluation  cycle  one.  The  standard  deviations  among 
the  classes  appeared  to  fluctuate,  although  they  were  all  of 
the  same  order  of  magnitude.  Comparison  of  mean,  median  and 


mode  for  each  class  revealed  that  four  of  the  classes 
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Table  1.  Pretest  Descriptive  Statistics  for  Evaluation  Cycle  One 


Class  Number 

Mean3 

Median 

Mode 

S.D. 

Distribution 

82-08  (N=12) 

43.56 

46 

46 

6.86 

Negative  Skew 

82-09  (N=10) 

43.01 

43 

None 

8.14 

Uniform^3 

82-10  (N=l 1 ) 

50.36 

50 

60 

9.08 

Positive  Skew 

82-11  (N=13) 

43.69 

40 

36 

6 .66 

Positive  Skew 

82-12  (N=10) 

39.30 

36 

36 

7.79 

Positive  Skew 

3 Overall  Mean  = 43.98.  b This  distribution  was  essentially 
rectangular. 
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exhibited  skewness  in  their  distributions,  while  one  appeared 
to  be  nearly  rectangular.  Based  on  mean  and  median 
statistics,  all  classes  appeared  similar  in  make-up  regarding 
their  levels  of  prior  knowledge,  and  that  the  overall  mean 
was  higher  than  the  guess  rate.  However,  the  pattern  of 
standard  deviations  indicates  consistently  variability  across 
classes.  An  explanation  for  this  relatively  large 

variability  may  be  that  some  of  the  material  presented  in  the 
course,  especially  that  dealing  with  techniques  of  selling, 
is  not  generic  to  Navy  Recruiting.  Entering  students 
appeared,  from  the  data,  to  possess  varying  amounts  of  such 
knowledge.  Overall,  the  assumption  that  entering  students 
would  score  at  a "guess  rate"  level  on  the  pretest  was  not 
empirically  verified. 

Actual  Observations  of  Transactions 

1.  The  spiral  course  sequence  was  verified  through  a correlation 
analysis  of  weekly  achievement  scores  (Table  2).  The 

bracketed  correlations  in  the  table  indicate  a moderate  to 
high  positive  correlation  (greater  than  0.40)  between  and 
among  various  weeks  of  the  revised  course.  For  example,  Week 
One  achievement  correlated  positively  with  Weeks  Two  and 
Five.  Week  Two  correlated  positively  with  Weeks  Three  and 
Five,  and  Week  Three  correlated  positively  with  Weeks  Four 
and  Five.  This  close  interrelationship  among  weekly 
achievement  scores  could  be  due  simply  to  the  fact  that 
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Table  2.  Specific  Pearson  Correlations  For  Evaluation  Cycle  One 


1 2 3 4 5 


1.  Week  1 Score 

2 . Week  2 Score 

[.68] 

3.  Week  3 Score 

[.43]  [.49] 

4.  Week  4 Score 

.30  .35  [.75] 

5.  Week  5 Score 

[.73]  [.84]  [.55]  [.43] 

NOTE:  Brackets  indicate  notable  correlations  (see  text) 
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brighter  students  learned  more  all  through  the  course. 
However,  if  it  is  assumed  that  classes  were  composed  of 
students  with  a random  mix  of  achievement  potential,  then  the 
correlations  suggest  that  the  spiral  sequence  was  maintained 
throughout  the  course.  That  is,  concept  attainment  in  the 
early  weeks  of  the  course  was  apparently  related  to 
achievement  in  later  weeks,  where  those  concepts  were 
reiterated  and  expanded. 

2.  On  the  whole,  instructional  procedural  fidelity  (actual 
classroom  transactions)  closely  followed  the  spiral  sequence 
embodied  in  the  course  schedules.  There  were,  however, 
several  minor  schedule  deviations.  Table  3 enumerates  the 
deviations  that  occurred  during  the  period  22  February 
through  26  February  1982: 

Week  1 : Student-Instructor  (S-I)  interactions  were  slow 

to  develop,  due  primarily  to  the  "newness"  of  the 
training  situation  for  the  students.  However,  the 
instructors  were  patient  and  prodding,  and  as.  the  week 
progressed,  S-I  interactions  increased.  Because  a 
workshop  did  not  take  place  on  the  fifth  day  of  the  week 
as  scheduled  (see  Table  3),  the  planned  interaction  of 
this  activity  was  not  observed. 

Week  2:  The  level  of  S-I  interaction  perceived  by  the 

investigator  was  higher  in  this  week  than  that  observed 
in  Week  1 . This  was  presumed  to  be  due  to  the  nature  of 
the  subject  matter  (sales  techniques)  as  well  as  the 


53 


TABLE  3. 
Date 

22  Feb 

23  Feb 

24  Feb 

26  Feb 

1 March 
5 March 

9 March 
10  March 

12  March 

15  March 

16  March 

19  March 

23  March 

25  March 

26  March 


Course  Deviations  (CD)  and  Unscheduled  Audio  Visual 
Requests  (UAV)  For  Evaluation  Cycle  One  (1982) 

Discrepancy 


Show  "Sales  Demo”  tape  for  Week  2 (UAV) 

Week  3 scheduled  "AAA  #3”  - never  called  for  (CD) 

Show  "AAA  //3"  for  Week  3 (UAV) 

Show  "AAA  #3"  for  Week  4 (UAV) 

Week  5 - Manuals  not  page  checked  (CD) 

Week  3 scheduled  "AAA  // 5&6"  - not  called  for  (CD) 
Week  3 - exam  given  in  morning  rather  than 

afternoon  (CD) 

Week  1 - workshop  not  conducted  (CD) 

Show  "Sales  Demo"  tape  for  Week  2 (UAV) 

Show  tape  "Phillipinos  in  Recruiting"  Week  3 (UAV) 
Week  1 - workshop  not  conducted  (CD) 

Show  "AAA  #4"  for  Week  3 (UAV) 

Week  5 - Administered  Recruiting  Manual  Exam  in 
morning  rather  than  afternoon  (CD) 

Week  1 - workshop  not  conducted  (CD) 

Show  tape  "Sales  Demo"  tape  for  Week  2 (UAV) 

Tape  sales  labs  for  Week  2 (UAV) 

Week  3 scheduled  "AAA  #3”  - never  called  for  (CD) 

Show  "AAA  #3"  for  Week  3 (UAV) 

Show  "AAA  #6"  for  Week  4 (UAV) 

Week  1 - workshop  not  conducted  (CD) 

Week  3 scheduled  "AAA  #3"  - never  called  for  (CD) 

Show  "AAA  #3"  for  Week  3 (UAV) 

Show  "AAA  #6"  for  Week  4 (UAV) 

Week  1 - workshop  not  conducted  (CD) 
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observed  intensity  and  vitality  of  the  instructors  in 
this  week  of  the  course. 

Week  3;  Interactions  between  students  and  instructor 
were  observed  to  be  less  intense  in  this  week  than  had 
been  observed  in  Week  2 . One  student  commented  that  it 
was  hard  to  keep  his  attitude  high,  but  that  the 
instructors  made  an  effort  to  boost  and  maintain  his 
attitude  throughout  the  week. 

Week  4 ; To  emphasize  the  spiral  nature  of  the  revised 
course  sequence,  there  was  need  for  content 
recapitulations,  scheduled  at  the  beginning  of  each  week 
of  the  course,  from  Week  2 through  Week  5.  The  intent 
of  these  recapitulations  was  to  help  students  merge 
concepts  learned  in  the  previous  weeks,  as  well  as 
those  to  be  taught  in  the  current  week,  into  an 
expanding  picture  of  recruiting.  Since  no  new  informa- 
tion was  scheduled  to  be  presented  in  Week  5,  the  most 
extensive  of  the  weekly  recapitulations  occurred  as  the 
first  scheduled  lesson  in  Week  4.  One  instructor, 
especially  knowledgeable  in  recruiting  practices, 
developed  a four-hour  group  discussion  lesson  designed 
so  students  could  fit  their  acquired  knowledge  of 
recruiting  concepts  into  a detailed  picture  of  the 
recruiting  process.  It  was  hoped  that  this  activity 
would  aid  students  in  internalizing  an  integrated  model 
of  recruiting.  The  rest  of  the  week's  instruction  was 
tied  into  this  integrated  model. 


55 


By 

the 

end  of  the  week, 

the 

total  recruiting 

process 

had 

been  presented  to 

the 

student.  This 

approach 

to 

Week  4 represented  a 

total  departure  from 

its  function  in  the  traditional  course.  Observations  of 
the  week  revealed  that  the  level  of  S-I  interaction  was 
dependent  upon  which  instructor  was  teaching.  As 
expected,  the  instructor  who  designed  the  week's 
activities  produced  the  greatest  amount  of  S-I  inter- 
action. It  was  also  observed  that  during  this  week 
other  instructors  reverted  to  the  traditional  passive- 
student  lecture.  It  became  apparent  through  such 
observations  that  several  instructors  were  not 

committed  to  the  spiraling  concept  as  embodied  in  the 
course  sequence. 

Week  5 : In  its  revised  form,  Week  5 was  designed  to 

present  no  new  information.  Rather,  it  was  to  be  a week 
wherein  students  would  apply  the  knowledge  they  had 
acquired  in  previous  weeks  to  both  actual  and  hypo- 
thetical recruiting  situations.  All  of  these  situations 
involved  a very  high  level  of  S-I  interaction. 

3.  The  relationship  between  course  objectives  and  job  tasks 
is  illustrated  in  Appendix  F.  Together  with  Appendixes 
D (Recruiter  Job  Task  Listing)  and  E (Course  Objectives 
List) , Appendix  F reveals  the  close  correspondence 
between  instructional  objectives  in  the  revised  course 
and  actual  field  recruiting  tasks.  This  correspondence 
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supports  the  contention  that  a high  degree  of  positive 
transfer  of  training  from  the  school  environment  to  the 
field  environment  was  planned  into  the  design  of 
the  course.  Since  no  record  of  a formal  task  analysis 
was  available  for  the  traditional  course,  no  comparison 
could  be  made  between  degrees  of  task-objective 
correspondence  in  the  two  courses. 

Graduates'  perceptions  of  the  course  were  analyzed 
to  determine  if  the  planned  correspondence  was  also 
perceived  by  graduates  of  the  course.  These  perceptions 
were  measured  via  a post-graduate  survey.  The 
investigator  conducted  a comparative  analysis  between 
surveys  from  graduates  who  had  attended  the  traditional 
course  and  surveys  from  graduates  of  the  revised 
course.  All  groups  were  surveyed  three  months  following 
their  graduation.  The  respondents  from  the  traditional 
course  accounted  for  21%  of  the  graduates  surveyed, 
while  those  from  the  revised  course  accounted  for  30%  of 
the  graduates  surveyed.  As  described  in  Chapter  III,  a 
five-point  response  scale  was  used.  The  ratings  on  the 
scale  were: 

1 - Task  not  performed;  eliminate  from  training 

2 - Training  more  than  adequate;  reduce  emphasis 

3 - Training  adequate  for  the  task 

4 - Training  less  than  adequate;  increase  emphasis 

5 - Training  requires  MUCH  more  emphasis  in  school 
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The  findings  of  this  comparative  analysis  are  shown 
in  Figure  6.  The  open  circles  in  the  figure  indicate 
the  mean  responses  of  graduates  from  the  traditional 
course.  The  solid  circles  indiate  the  mean  responses  of 
graduates  from  the  revised  course.  Numbers  along  the 
bottom  of  the  graph  (1  through  36)  indicate  questions  on 
the  graduates'  survey  (cf.  Appendix  C).  The  scales  on 
either  side  of  the  graph  indicate  the  response  values. 
Since  the  survey  results  of  each  group  were  arithmetic- 
ally averaged,  it  was  possible  to  obtain  a virtually 
continuous  range  of  ratings  from  1.0  to  5.0,  rather  than 
only  five  discrete  values. 

There  is  a considerably  higher  degree  of 
variability  in  the  percieved  applicability  of  the 
training  received  in  the  traditional  course  (mean 
variance  = 0.17)  compared  to  that  in  the  revised  course 
(mean  variance  = 0.03).  That  is,  the  perceptions  of 
graduates  from  the  revised  course  appear  to  deviate  much 
less  about  the  3.0  reference  level  (dashed  line)  than  do 
perceptions  of  graduates  from  the  traditional  course. 
This  lends  support  to  the  contention  that  the  revised 
course  may  have  a higher  perceived  fidelity  to 
real-world  recruiting,  thus  producing  a more  effective 
positive  transfer  of  training,  than  did  the  traditional 


course. 
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Actual  Observations  of  Outcomes 

1.  The  academic  attrition  rate  in  the  five  traditional  classes 
was  four  out  of  110  students  (3.6%).  During  the  first  evalu- 
ation period  of  the  revised  course,  seven  academic  attritions 
out  of  109  students  (6.4%)  were  noted.  The  specific  reason 
for  the  increase  in  attrition  in  the  first  evaluation  period 
is  unknown.  Some  plausible  reasons  for  this  increase  are, 
first,  the  rise  in  the  attrition  rate  in  the  first  evaluation 
period  of  the  revised  course  may  be  due  to  a possible 
increase  in  the  difficulty  of  the  course  caused  by  the  new 
course  sequence;  second,  the  increase  could  be  due  to  the 
possibility  that  poor  attrition  record  keeping  in  the  tradi- 
tional course  produced  lower  than  actual  figures,  thus  lead- 
ing to  an  apparently  deflated  attrition  rate  in  the 
traditional  course;  third,  the  increase  could  be  due  to  a 
chance  variation  caused  by  the  relatively  small  population 
size.  A fourth  possibility  could  be  a combination  of  all  of 
the  above  possibilities.  A fifth  possibility  relates  to  the 
number  of  schedule  deviations  discovered  throughout  the 
course  (Table  3).  These  data  suggest  some  degree  of  reduc- 
tion in  the  instructors'  committment  to  the  spiral  sequence 
of  the  revised  course.  Since  such  a reduction  was  found  to 
occur,  it  may  have  impaired  student  comprehension  of  the 
intricate  relationships  in  the  course  content.  If  the 
instructors  exacerbated  this  condition  by  ignoring  the  spiral 
sequence  of  the  course,  this  could  have  easily  led  to  the 


rise  in  the  rate  of  academic  attrition  observed  in  the  data. 
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2.  Data  coacerning  academic  roll-backs  were  not  available  for 
the  traditional  course,  since  they  had  not  been  maintained  by 
any  department  at  the  school  prior  to  the  beginning  of  the 
revised  course.  Thus,  a comparison  between  the  traditional 
and  revised  courses  on  this  dimension  was  not  possible. 

Results  of  Evaluation  Cycle  Two 

Actual  Observations  of  Antecedent 

1.  For  this  evaluation  cycle,  the  course  pretest  was 
discontinued  due  to  the  non-availability  of  a suitable  time 
slot  during  the  new-student  orientation  period.  Time  that 
had  been  allotted  for  administration  of  the  pretest  during 
the  first  evaluation  cycle  was  re-allocated  in  the  current 
cycle  for  other  types  of  student  testing. 

Actual  Observations  of  Transactions 

1.  During  the  second  evaluation  cycle,  correlations  among  weekly 
achievement  scores  were  found  to  be  slightly  lower  than  those 
of  the  first  evaluation  period.  Table  4 shows  comparative 
achievement  correlations  for  the  first  and  second  evaluation 
cycles.  As  evidenced  in  the  data,  the  higher  correlations 
reported  in  the  first  evaluation  cycle  are  generally 
associated  with  lower  correlations  in  the  second  cycle,  while 
the  more  moderate  correlations  from  cycle  1 appear  to  have 
remained  fairly  consistent  with  those  of  cycle  2.  Although 
these  intra-course  correlations  generally  decreased  in  the 
second  evaluation  period,  they  remained  at  a moderate  level. 
Chance  variability  cannot  be  ruled  out  as  an  explanation  for 
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Table  4.  Comparative  Correlations  Between  Evaluation 
Cycles  One  and  Two 

1 2 3 4 5 


1 . Week  1 

2 . Week  2 

.68/. 30 

3 . Week  3 

.43/. 54  .49/. 40 

4.  Week  4 

.30/. 48  .34/. 29  .75/. 59 

5 • Week  5 

.73/. 41  .84/. 28  .55/. 52  .43/. 47 

NOTE:  Correlations  reported  in  the  table  are  presented  in  the 

following  form:  Cycle  One/Cycle  Two 
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these  correlation  shifts,  because  correlations  from  small 
sample  sizes  are  exceedingly  unstable.  Assuming  chance 
variability  as  the  cause  for  the  changes  in  correlation 
coefficients  leads  the  investigator  to  conclude  that  content 
and  minor  sequence  changes  made  to  the  course  prior  to  the 
second  evaluation  period  apparently  had  no  measurable 
positive  or  adverse  effects  on  the  integrity  of  the  spiral 
sequence  originally  designed  into  the  revised  course. 

Specific  topics  dealing  with  sales  techniques,  pre- 
enlistment kit  preparation  and  production  analysis,  which  had 
been  spiraled  through  every  week  of  the  course  in  the  first 
evaluation  cycle,  were  shifted  in  such  a manner  that  they 
were  no  longer  spiraled  through  every  week.  This  was  a 
school  administrative  action  which  was  beyond  the  control  of 
the  investigator. 

The  computer  program  that  derived  the  original  sequence 
for  the  revised  course  was  used  in  the  second  evaluation 
cycle  to  determine  the  degree  to  which  the  changes  made  to 
the  curriculum  detracted  from  the  original  spiral  design. 
Analysis  revealed  that  no  major  breaks  in  the  logical 
sequence  of  the  course  content  were  introduced  by  these 
changes.  However,  several  strong  transitions  between  related 
topics  were  weakened  as  the  topics  were  spaced  farther  apart 
in  the  course.  Considering  the  results  shown  in  Table  4, 
these  sequence  changes  may  or  may  not  have  had  an  effect  on 
the  strength  of  the  correlations  between  student  weekly 


examination  scores. 
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2.  The  number  of  course  deviations  occurring  in  the  second 
evaluation  period  decreased  dramatically  from  the  level  found 
during  the  first  evaluation  cycle.  Table  5 enumerates  the 
schedule  deviations  recorded  during  the  second  evaluation 
period,  18  April  1983  to  15  July  1983.  Observations  of 
student-instructor  (S-I)  interactions  in  each  week  of  the 
course  revealed  the  following: 

Week  1 : As  was  observed  in  the  previous  evaluation  cycle, 

S-I  interactions  were  slow  to  develop  throughout  the  week. 
The  student  workshop,  previously  scheduled  on  the  fifth  day 
of  the  week,  had  been  discontinued  in  the  interim  between 
evaluation  cycles.  This  eliminated  a potentially  effective 
avenue  for  S-I  interactions  in  the  week.  The  norm  for  the 
week  was  lecture,  which  resulted  in  a meager  level  of  S-I 
interaction.  Students  received  feedback  on  their  progress 
via  three  morning  quizzes  on  the  second,  third  and  forth  days 
of  the  week.  These  quizzes  covered  material  presented  in 
previous  lessons,  as  well  as  material  covered  in  nightly 
homework  assignments. 

Week  2:  There  was  no  observed  change  in  either  the  level  of 

S-I  interaction  or  in  the  content  presented  during  the  week 
compared  with  that  observed  in  the  first  evaluation  period. 
Week  3:  Observations  revealed  no  major  changes  in  this 

week's  transactions  from  those  reported  in  the  first 


evaluation  period. 
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Table  5.  Course  Deviations  (CD)  and  Unscheduled  Audio  Visual 
Reqests  (UAV ) for  Evaluation  Cycle  Two.  (1983) 


Date 

Discrepancy 

29  April 

Show  "Navy  Campus"  film  for  Week  Three  (UAV) 

3 May 

Show  film  on  "Drugs  and  Driving"  for  Week  5 (UAV) 

9 May 

Week  5 "Time  Management"  film  - never  called  for  (CD) 

13  May 

Show  "Navy  Campus"  film  for  Week  Four  (UAV) 

10  June 

Week  3 film  "One  for  the  Road"  - never  called  for  (CD) 
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Week  4:  The  primary  function  of  this  week  was  to  present 

detailed  operations  of  recruiting  tracking  and  production 
analysis  systems  (the  theory  of  which  was  presented  in  Week 
3).  Observations  by  the  investigator  revealed  a level  of 
S-I  interaction  relatively  higher  than  that  found  in  Week  3. 

- Presumably,  this  was  due  to  the  fact  that  students  had  been 
exposed  to  these  concepts  in  the  preceeding  week. 

Week  5:  Since  the  inception  of  the  revised  course,  this 

week  was  planned  to  be  an  applications  week  - a time  designed 
for  students  to  apply  the  concepts  learned  previously  in  the 
course  in  both  hypothetical  and  actual  recruiting  situa- 
tions. The  instructor-supervisor  of  Week  5,  appointed 
shortly  before  the  beginning  of  the  second  evaluation 
period,  developed  a series  of  application  workshops  for 
the  week.  The  implementation  of  these  workshops  created  a 
level  of  S-I  interaction,  perceived  by  the  investigator,  to 
be  equal  to  that  observed  in  Week  2 , and  higher  than  that 
observed  for  Week  5 during  the  previous  evaluation  period. 

3.  Because  no  instructional  objectives  were  added  to  or  deleted 
from  the  course  between  the  first  two  evaluation  cycles, 
there  was  no  change  in  the  theoretical  correspondence  between 
course  objectives  and  field  recruiting  tasks.  To  put  this  in 
perspective,  the  correspondence  ratings  perceived  by  the 
course  graduates  were  analyzed.  Figure  7 presents  a compara- 
tive graphic  analysis  of  graduate  perceptions  in  the  first 


two  evaluation  cycles  of  the  revised  course. 


The  scales 
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Figure  7 . Comparison  of  Graduate  Perceptions  of  Course  Adequacy 
Between  the  First  and  Second  Evaluation  Cycles  of  the  Revised  Course 
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on  the  bottom  and  sides  of  the  graph  have  the  same  meaning  as 
in  the  graph  of  Figure  6.  The  open  circles  represent  the 
mean  of  responses  for  graduates  from  the  first  evaluation 
cycle,  while  the  solid  circles  represent  the  mean  of 
responses  for  graduates  of  the  second  cycle.  The  mean 
variance  in  the  perceived  applicability  of  school  training  by 
graduates  of  the  first  cycle  was  0.03,  whereas  the  mean 
variance  in  responses  from  the  second  cycle  was  0.05.  The 
slight  increase  in  mean  variance  in  the  second  cycle  was 
still  considerably  less  than  that  for  the  traditional  course, 
whose  mean  variance  was  0.17  (cf.  Figure  6).  This  indicates 
that  graduates  of  the  second  evaluation  cycle  perceived 
overall  training  fidelity  to  be  nearly  the  same  as  that  of 
the  first  cycle.  Together,  these  two  findings  support  the 
contention  that  the  revised  course  had  a higher  perceived 
fidelity  than  did  the  traditional  course. 

Actual  Observation  Outcomes 

1.  During  the  second  evaluation  period,  academic  attrition 
accounted  for  14  losses  out  of  222  students  (6.3%).  The 
slight  decrease  in  academic  attrition  rate  between  the  second 
cycle  and  the  first,  with  a 6.4%  reported  rate,  indicates 
that  the  revised  course  remained  essentially  stable  over  the 
two  cycles  with  regard  to  academic  attrition  rate.  Yet,  this 
rate  was  still  higher  in  the  revised  course  than  that 
determined  for  the  traditional  course. 

The  academic  roll-back  rate  for  the  second  cycle  was  computed 
to  be  15%  (34  roll-backs  out  of  222  students  enrolled). 


2. 
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Compared  with  the  computed  roll-back  rate  for  the  first 
cycle,  28%  (30  out  of  109),  it  was  evident  that  the  second 
evaluation  cycle  experienced  a dramatic  decrease  in  intra- 
course failures  over  the  first  cycle.  Coupled  with  the 
slight  decrease  in  academic  attrition  rate,  this  finding 
lends  support  to  the  contention  that  instruction  in  the 
second  cycle  led  to  greater  student  comprehension  of  material 
than  in  the  first  cycle.  This  finding  also  implies  that  the 
Hawthorne  effect  may  not  have  had  as  strong  an  effect  on  the 
revised  course  as  was  initially  anticipated. 

In  summary  of  the  first  two  evaluation  cycles,  the 
predominant  findings  were: 

1.  Intra-course  achievement  correlations  were  generally 
lower  than  those  of  the  first  cycle, 

2.  Academic  attrition  rate  decreased  slightly  from  its 
level  in  the  first  cycle, 

3.  Academic  roll-back  rate  decreased  sharply  in  the 
second  cycle,  and 

4.  Recorded  occurrences  of  schedule  deviations  declined 
sharply  in  the  second  cycle. 

Finding  1 implies  that  the  spiral  nature  of  the  course 
sequence  had  deteriorated  somewhat  in  the  second  cycle.  Three 
possible  reasons  for  this  finding  are:  1)  the  student  groups  in 

these  cycles  were  different,  and  these  differences  accounted  for 
the  decrease,  2)  sequence  changes  imposed  on  the  course  prior  to 
the  second  cycle  accounted  for  the  variance  in  the  correlations. 
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or  3)  a chance  variation.  It  is  not  possible  with  the  available 
data  to  refute  either  the  possibility  that  the  two  populations 
were  different  academically,  or  that  variations  were  due  strictly 
to  chance. 

However,  regarding  the  second  explanation,  the  sequence 
changes  that  were  made  were  apparently  more  functional  for  the 
instructor  staff  as  evidenced  by  the  sharp  decline  in  schedule 
deviations  (finding  4).  This  finding  suggests  that  instructors 
could  more  easily  teach  the  modified  spiral  seuqence  and,  as  a 
result,  help  the  student  relate  to  it  more  easily.  Early  in  the 
course's  development,  it  was  presumed  that  if  students  understood 
the  spiral  nature  of  the  course  material,  they  would  have  a 
greater  probability  of  internalizing  the  concepts  presented;  thus, 
they  should  perform  better  in  school.  Finding  3 supports  this 
supposition.  It  seems  reasonable  to  suspect  that  schedule  changes 
instituted  in  the  course  accounted  for  at  least  a portion  of  the 
variance  in  academic  correlations. 

Results  of  Evaluation  Cycle  Three 
Actual  Observations  of  the  Antecedent 

1.  The  course  pretest  was  not  used  during  this  evaluation  cycle 
for  the  same  reason  described  in  evaluation  cycle  two,  i.e. 
insufficient  time  was  available  to  administer  the  pretest  due 
to  other  administrative  requirements. 

Actual  Observations  of  Transactions 

1.  Table  6 illustrates  comparative  correlation  coefficients 
among  student  achievement  scores  in  each  week  of  the  course, 
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Table  6.  Comparative  Correlation  Coefficients  Among  Three 

Evaluation  Cycles 


1 2 3 4 5 


Correlations  reported  in  the  table  are  presented  in  the 
following  form:  Cycle  One/Cycle  Two/Cycle  Three.  Numbers  in 
parentheses  are  mean  correlation  values  for  each  triplet  (see 
text) . 


NOTE: 
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from  all  three  evaluation  cycles.  To  make  these  correlations 
more  useful  for  comparative  purposes,  they  were  first 
normalized  via  a Fisher  ' z'  transformation,  then,  averaged  to 
yield  a mean,  and  finally  transformed  back  into  Pearson 
correlations.  These  means  are  displayed  in  parentheses 
directly  beneath  each  triplet  in  the  table.  The  means 
display  a moderate  correlation  between  student  academic 
performance  in  various  weeks  of  the  course.  The  data  appear 
to  suggest  a pattern  of  spiraling  among  the  weeks  that 
matches  the  planned  spiral  concept. 

For  example,  the  highest  mean  correlation  in  the  table, 
that  between  Weeks  3 and  4,  was  expected  due  to  the  fact  that 
Week  4 presented  applications  of  the  Activity  Analysis 
content  presented  in  Week  3.  It  was  expected  that  students 
who  did  well,  academically,  in  Week  3 would  also  do  well  in 
Week  4.  The  data,  in  this  case,  appear  to  substantiate  the 
expectation.  However,  the  correlations  are  not  extremely 
strong,  considering  the  large  degree  of  spiraling  designed 
into  the  course.  Because  the  cycle  three  correlations  do  not 
establish  consistent  trends,  they  seem  to  support  an 
explanation  of  chance  variability.  A possible  reason  for 
this  can  be  found  in  the  research  of  Bloom  (1976)  and  others 
who  have  shown  that  a great  many  factors  (not  investigated  in 
this  study)  affect  variance  in  academic  achievement,  while 
instructional  sequence  accounts  for  only  a portion  of  this 


variance. 


72 


2.  Schedule  deviations  recorded  during  the  third  evaluation 
cycle  were  observed  to  be  at  their  lowest  levels  since  the 
beginning  of  the  study.  Table  7 enumerates  the  deviations 
recorded  in  the  school's  Audio  Visual  Usage  records  for  the 
period  8 July  1983  to  30  September  1983.  Observations  of 
student-instructor  (S-I)  interactions  throughout  the  course 
revealed  the  following: 

Week  1:  As  observed  in  the  previous  two  evaluation  cycles, 

S-I  interactions  were  slow  to  develop.  Students  in  this 
cycle  received  feedback  on  their  progress  via  three  morning 
quizzes  on  the  second,  third  and  forth  days  of  the  week,  as 
in  the  previous  cycle.  These  quizzes  were  identical  to  those 
administered  to  students  during  the  second  cycle. 

Week  2:  There  was  no  observed  change  in  either  the  level  of 

S-I  interaction  or  in  the  content  presented  in  the  week  from 
that  observed  in  either  the  first  or  second  evaluation 
period . 

Week  3:  Classroom  transactions  were  essentially  unchanged 

compared  to  those  observed  in  the  previous  cycle. 

Week  4:  As  in  the  second  evaluation  cycle,  the  primary 

function  of  this  week  continued  to  be  a presentation  of  the 
detailed  operations  of  recruiting  tracking  and  production 
analysis  systems.  The  high  level  of  S-I  interaction 

observed  in  the  second  evaluation  cycle  was  still  evident  in 


the  third  cycle. 
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Table  7.  Course  Deviations  (CD)  and  Unscheduled  Audio  Visual 
Requests  (UAV)  in  Evaluation  Cycle  Three  (1983). 


Date 

Discrepancy 

14  July 

Show  "Sales  Demo"  tape  for  Week  2 (UAV) 

21  July 

Show  "Sales  Demo"  tape  for  Week  2 (UAV) 

8 Aug 

Request  to  tape  Week  5 Skill  Labs  (UAV) 

2 Sept 

Week  1 film  "Navy  Campus"  - never  called  for  (CD) 
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Week  5:  As  stated  earlier  in  this  chapter,  this  week  was 

planned  to  be  solely  an  applications  week.  However, 
counter  to  this  initial  intent,  some  new  instruction  was 
inserted  into  the  week.  This  reduced  the  amount  of  time 
available  during  the  week  for  student  application  labs,  as 
compared  with  that  observed  in  evaluation  cycle  two. 

3.  As  was  the  case  in  the  second  evaluation  cycle,  no 
instructional  objectives  were  added  to  or  deleted  from  the 
course  either  before  or  during  the  third  evaluation  cycle. 
Thus,  again  there  was  no  change  in  the  theoretical 
correspondence  between  course  objectives  and  field 
recruiting  tasks. 

The  pattern  revealed  in  graduates'  perceptions  of  this 
correspondence,  however,  did  change  from  that  of  the  second 
cycle.  Figure  8 shows  a comparative  graphic  analysis  of 
graduates'  perceptions  in  the  second  and  third  evaluation 
cycles  of  the  revised  course.  Table  8 displays  mean 
perception  ratings  of  graduates  across  the  entire  study. 
The  scales  on  the  bottom  and  sides  of  the  graph  have  the 
same  meaning  as  for  the  graphs  of  Figures  6 and  7.  Open 
circles  represent  the  mean  responses  of  graduates  from  the 
second  evaluation  cycle,  while  the  solid  circles  represent 
those  for  the  third  cycle.  Mean  variance  in  graduates' 
perceived  applicability  of  school  training  from  the  second 
cycle  was  0.05;  while  the  mean  variance  in  the  third  cycle 
was  0.06.  Mean  variance  increased  from  the  second  to  the 


third  cycle,  just  as  it  had  from  the  first  cycle  to  the 
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Figure  8.  Comparison  of  Graduate  Perceptions  of  Course  Adequacy 
Between  the  Second  and  Third  Evaluation  Cycles  of  the  Revised  Cour; 
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Table  8.  Mean  Perception  Ratings  of  Course  Adequacy  by 
Graduates  from  the  Traditional  and  Revised  Courses. 


I tern 

Traditional 

Course 

I 

Revised  Course  Cycles 
II 

III 

1 

3.0 

2.8 

3.0 

3.1 

2 

3.0 

2.9 

3.1 

3.3 

3 

2.7 

2.8 

3.0 

2.9 

4 

3.3 

2.9 

3.0 

3.2 

5 

3.6 

3.2 

3.0 

3.0 

6 

2.3 

3.1 

3.2 

3.2 

7 

2.3 

3.0 

3.2 

3.2 

8 

2.3 

3.1 

3.2 

3.4 

9 

2.3 

3.1 

3.3 

3.2 

10 

2.7 

3.1 

3.3 

3.4 

11 

2.3 

3.1 

3.2 

3.5 

12 

2.7 

3.1 

3.0 

2.9 

13 

2.7 

3.2 

3.2 

3.2 

14 

3.6 

3.3 

2.5 

2.8 

15 

2.3 

3.0 

2.7 

2.9 

16 

3.0 

3.1 

3.4 

3.3 

17 

3.0 

2.7 

2.9 

3.2 

18 

3.0 

2.7 

3.1 

3.5 

19 

3.6 

3.3 

2.9 

3.2 

20 

2.3 

3.0 

3.0 

3.0 

21 

3.0 

3.2 

2.9 

2.8 

22 

3.0 

3.2 

2.9 

2.8 

23 

3.3 

3.2 

3.3 

3.2 

24 

3.3 

3.0 

2.8 

2.8 

25 

3.3 

3.0 

2.7 

2.7 

26 

3.3 

3.1 

3.0 

3.0 

27 

2.3 

2.9 

3.1 

3.2 

28 

2.3 

3.0 

3.2 

3.1 

29 

2.7 

3.2 

2.8 

2.7 

30 

2.7 

3.3 

3.0 

2.8 

31 

2.7 

3.1 

2.9 

2.7 

32 

3.0 

3.2 

2.8 

2.9 

33 

2.7 

3.3 

3.1 

3.1 

34 

2.3 

2.8 

3.4 

3.5 

35 

2.7 

2.9 

3.3 

3.5 

36 

2.7 

2.9 

3.3 

3.2 
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second.  Despite  this  increase,  however,  the  overall 
variance  was  still  lower  than  that  observed  for  the 
traditional  course.  The  perceived  applicability  of 
training  to  real-world  recruiting  was,  apparently, 
consistently  greater  for  the  revised  course  than  for  the 
traditional  course. 

Actual  Observations  of  Outcomes 

1.  During  the  third  evaluaion  cycle,  academic  attrition 
accounted  for  five  losses  out  of  100  students  enrolled 
(5.0%).  Considered  together,  the  attrition  rates  for  all 
three  evaluation  cycles  represented  a consistent  declining 
trend  throughout  the  entire  evaluation  period  of  the 
revised  course.  This  decline  is  illustrated  graphically  in 
Figure  9 (solid  line).  As  mentioned  earlier  in  this 
chapter,  because  historical  attrition  data  were  not 
available  on  a class-by-class  basis  for  the  traditional 
course,  no  relative  comparisons  between  the  two  courses 
were  possible  on  this  dimension.  The  only  available 
attrition  rate  for  the  traditional  course  was  that  quoted 
in  the  findings  for  evaluation  cycle  one  (3.6%),  which  was 
not  assumed  to  be  representative  of  the  traditional 
course.  Two  things  can  be  said  in  favor  of  the  revised 

course.  First,  a steady  decline  in  academic  attrition 
rates  was  observed  over  the  duration  of  the  evaluation 
study  (cf.  Figure  9).  Second,  all  academic  attrition 

rates,  though  higher  than  that  in  the  traditional  course, 
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Figure  9.  A Trend  Line  Plot  of  Academic  Attrition  Rates 
(solid  line)  Versus  Roll-back  Rates  (dashed  line) 
for  the  Revised  Curriculum 
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were  considerably  less  than  the  25%  ceiling  established  by 
the  Navy  Recruiting  Headquarters  Command. 

2.  The  academic  roll*-back  rate  for  the  third  cycle,  computed 
to  be  12%  (12  out  of  100  students),  maintained  a declining 
trend  with  the  computed  roll-back  rates  for  the  first  and 
second  cycles  (28%  and  15%  respectively).  As  was  the  case 
with  attrition  rates,  academic  roll-back  rates  exhibited  a 
consistently  declining  trend  across  all  evaluation  cycles 
(Figure  9,  dashed  line).  This  indicates  that,  over  the 
duration  of  the  evaluation  study,  instruction  under  the 
revised  course  sequence  continually  improved  in  producing 
increased  student  comprehension  of  material.  However,  this 
is  not  sufficient  to  indicate  the  superiority  of  the 
revised  course  over  the  traditional  course,  since  no 
comparative  roll-back  data  were  available  for  the  latter 
with  which  to  make  a definitive  decision.  The  best  that 
can  be  said  is  that  the  revised  course  sequence  appears  to 
be  effective  in  long-term  reduction  of  within-course 
academic  failures  (roll-backs). 


CHAPTER  V 


SUMMARY , CONCLUSIONS  AND  RECOMMENDATIONS 

The  remaining  column  in  the  Judgement  Matrix  of  the 
Countenance  Model  is  composed  of  judgements  about  the  OBSERVATIONS 
(findings)  based  on  the  pre-established  STANDARDS  enumerated  in 
the  previous  chapter.  These  judgements  form  the  basis  for  the 
conclusions  and  recommendations  contained  in  this  chapter. 

The  purpose  of  the  study  was  to  evaluate  several  aspects  of  a 


revised 

course 

in 

recruiting 

techniques  whose  instructional 

sequence 

was 

based 

on 

Bruner' 

s spiraling 

paradigm. 

and  was 

generated 

with 

the 

aid 

of  an 

investigator- 

-developed 

computer 

program  employing  the  Interpretive  Structural  Modeling  (ISM) 
methodology.  The  revised  course  sequence  was  compared  with  a 
previous  version  of  the  same  course  which  had  been  sequenced 
according  to  a manual  Gagne- type  methodology.  Although  not 
designed  to  determine  the  superiority  of  either  sequencing 
methodology,  this  evaluation  was  designed  to  illustrate  the 
efficacy  of  an  I SM-developed  curriculum. 

Summary  of  Findings 

1.  Is  there  a difference  in  academic  attrition  rates  between 
the  traditional  and  revised  courses? 

Average  academic  attrition  from  the  revised  course  (5.9%)  was 
greater  than  that  for  the  traditional  course  (3.6%).  This  does 
not  mean,  however,  that  the  traditional  course  was  necessarily 
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more  effective  than  the  revised  course,  because  there  was  no 
historical  documentation  specifying  the  criteria  used  in  the 
traditional  course's  academic  attrition  process.  The  rate  in  the 
traditional  course  could,  therefore,  have  been  an  anomoly  produced 
by  the  particular  (unknown)  attrition  criteria  used.  If  it  is 
assumed  that  this  attrition  rate  were  not  anomolous,  then  the 

observed  increase  in  attrition  rate  in  the  revised  course,  though 
surprising,  is  less  disturbing  due  to  the  downward  trend  in  the 
revised  course's  attrition  rate  across  all  evaluation  cycles,  as 
addressed  in  the  following  finding. 

2.  Is  there  evidence  of  either  stability  or  consistent 

decline  in  academic  attrition  rates  across  several 
independent  observations  of  the  revised  course? 

Attrition  rate  in  the  revised  course  consistently  declined 
throughout  the  evaluation  period  (Figure  9).  This  observed  trend 
indicates  that  the  revised  course  demonstrated  long-term 
effectiveness  in  the  ability  to  decrease  academic  attrition  rate. 

Because  the  three  evaluation  cycles  were  implemented  during 
the  late  winter,  early  spring  and  late  summer  periods,  it  can  be 
argued  that  the  observed  decline  in  academic  attrition  is  a result 
of  some  type  of  seasonal  variation  in  the  student  population 

attending  the  course  during  each  evaluation  cycle.  The  investi- 
gator does  not  consider  this  to  be  a valid  concern,  however, 
because  there  is  no  statistical  basis  for  such  variation  in 

specialized  Navy  training  schools.  This  is  an  area  for  future 


research. 
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On  the  basis  of  this  steadily  declining  trend  across  the 
year-and-a-half-long  observation  period,  the  revised  course 
appears  to  possess  the  capability  of  reaching  a low  level  of 
attrition.  Even  if  the  traditional  course's  observed  attrition 

rate  were  an  anoraoly,  the  investigator  still  concludes  that  the 
revised  course  was  at  least  as  effective  as  the  traditional  course 
based  on  reduction  in  attrition. 

3.  How  do  graduates'  perceptions  of  the  applicability  of 
training  to  real-world  recruiting  requirements  differ  in 
the  traditional  course  compared  to  those  in  the  revised 
course? 

Analysis  of  graduates'  survey  results  from  both  the 
traditional  and  revised  courses  (Figures  6,  7 and  8 respectively), 
indicated  that  graduates  of  the  revised  course,  on  average, 
perceived  a higher  level  of  overall  training  fidelity  than  did 
graduates  of  the  traditional  course.  In  and  of  itself,  this 
finding  suggests  that  the  revised  course  provided  a greater 
perceived  realism  in  training  than  the  traditional  course.  If,  as 
Gagne  implies,  such  realism  leads  to  more  positive  transfer  of 
learned  concepts  from  the  school  environment  to  the  real  world, 
then  it  appears  that  the  revised  course  sequence  was  more 
effective  on  this  dimension  than  the  traditional  course.  To 
provide  additional  support  for  the  effectiveness  of  the  revised 
course  on  this  dimension,  future  studies  should  collect  data  on 
course  graduates  regarding  their  success  in  recruiting  (known  as 
production  per  recruiter  statistics),  the  demographic  location  of 
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their  recruiting  facilities  and  their  Recruiting  District's  style 
of  management.  It  has  been  suggested  that  variables  such  as  these 
may  affect  the  return  rate  of  graduates'  questionnaires  (Dyer  et 
al.,  1975).  It  is  suggested  here  that  these  factors  could  also 
potentially  affect  graduates'  perceptions  of  resident  school 
training  which,  if  left  unaccounted  for,  could  produce  a 
misleading  view  of  the  perceived  fidelity  of  the  course. 

4 . Is  there  evidence  of  a change  in  the  level  of  procedural 
fidelity  regarding  classroom  instruction  in  the  revised 
course  over  the  duration  of  the  study? 

Analysis  of  course  deviations  and  unscheduled  audio-visual 
requests  for  each  of  the  three  evaluation  cycles  (Tables  3,  5 and 
7 respectively)  displayed  a sharply  declining  trend  throughout  the 
study.  The  trend  was  found  to  be  in  the  direction  postulated  in 
the  standards  for  these  transactions.  This  finding  provides 
evidence  to  support  the  contention  that  the  instructor  staff 
gradually  adapted  their  instructional  processes  to  the  revised 
course  sequence.  That  the  decline  appeared  to  occur  in  concert 
with  the  changes  initiated  in  the  course  sequence  suggests  a 
relationship.  Whether  or  not  such  a relationship  actually  existed 
cannot  be  verified  from  the  data  collected.  Nevertheless,  the 
fact  that  sequence  deviations  did  decline  is  important  because  the 
staff's  committment  to  the  revised  sequence  was  originally  an 
unknown  factor  having  the  potential  to  seriously  impair  the 
academic  effectiveness  of  the  course.  Based  on  this  finding,  the 
investigator  concluded  that  staff  inertia  against  the  revised 
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course  sequence  was  not  a long-term  problem,  and  therefore,  is 
discounted  as  a potentially  perturbing  effect  on  the  other 
findings  of  the  study.  Future  evaluations  of  the  course  will, 
therefore,  not  have  to  contend  with  this  potential  problem  unless 
another  massive  course  revision  is  made. 

5.  Is  there  a difference  in  academic  roll-back  rates  across 
several  independent  observations  of  the  revised  course? 

Just  as  with  attrition  rates,  the  roll-back  rates  from  the 
revised  course  displayed  a strong  decreasing  trend.  Both  the 
roll-back  rate  and  attrition  rate  decreased  in  concert  with  one 
another  throughout  the  study  (cf.  Figure  9).  The  correlation 
between  attrition  rate  and  roll-back  rate  across  all  observation 
periods  is  approximately  .30.  This  suggests  only  a slight 
interrelationship  between  the  decline  in  the  two  rates.  Thus, 
because  both  rates  did  decrease  in  a more  or  less  parallel 
fashion,  and  because  the  decreases  were  not  strongly  dependent  on 
one  another,  there  is  support  for  the  contention  that  the  revised 
course  was  effective  in  reducing  both  attrition  rate  and  roll-back 
rate,  independently. 

Conclusion 

This  study  has  provided  evidence  that  the  ISM-developed 
(revised)  course  sequence  was  at  least  as  effective  as  the 
Gagne-type  (traditional)  course  sequence  in  reducing  attrition  and 
within-course  failure  rates,  and  in  raising  course  graduates' 
perceptions  of  the  fidelity  of  training  received  in  school.  These 
findings  support  the  contention  that  ISM  is  a viable  alternative 
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methodology  for  developing  reliable  and  easily  manipulable 
instructional  hierarchies  and  course  sequences  in  the  realm  of 
competency-based  instruction.  Pending  experimental  findings  in 
favor  of  the  ISM  methodology  as  applied  to  instructional  hierarchy 
development,  it  appears  that  ISM  possesses  the  potential  to 
facilitate  the  hierarchy  development  process,  and,  to  this  extent, 
may  assist  in  increasing  the  amount  and  quality  of  future  research 
related  to  hierarchical  learning. 

Recommendations 

Further  research  should  be  conducted  to  determine  the  effica- 
cy of  ISM  methodology  in  developing  instructional  hierarchies  and 
sequences.  Such  research  should  involve  comparisons  between  an 
experimental  course  developed  via  ISM  and  a control  version  of  the 
same  course  developed  with  the  Gagne  technique.  Based  on  the 
research  of  Bloom  and  others,  there  appears  to  be  a link  between 
sequence  and  academic  achievement.  There  is  a need,  therefore,  to 
conduct  research  on  ISM  developed  instructional  sequences  for 
non-competency-based  courses,  where  achievement  scores  are  more 
variable  than  in  competency-based  courses. 

Because  ISM  is  designed  to  resolve  complexities  in  a system 
of  interacting  elements,  it  is  interesting  to  speculate  whether 
ISM  is  capable  of  resolving  the  complexities  inherent  in  the 
structure  of  disciplines  of  knowledge.  It  is  apparent,  for 
example,  that  most  disciplines  are  interrelated  to  one  degree  or 
another.  However,  the  complexity  of  the  whole  issue  of  the 
structure  of  knowledge  has  hindered  major  attempts  to  quantify 
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those  interrelationships.  Given  that  ISM  possesses  the  capability 
of  resolving  complex  systems  into  more  simple  arrangements  of 
their  constituent  components,  it  is  tempting  to  speculate  on  the 
possibility  that,  with  the  aid  of  ISM  techniques,  research  could 
ensue  in  this  area.  At  present,  it  is  apparent  to  this 
investigator  that  such  an  undertaking  would  involve  an 
interdisciplinary  approach  involving  the  best  minds  in  each  field 
of  knowledge.  Perhaps  research  such  as  this  will  reveal  the 
findings  necessary  for  development  of  a truly  comprehensive  theory 
of  instruction. 


Appendix  A 


AN  APPLICATION  OF  GAGNE'S  INSTRUCTIONAL  SEQUENCING  TECHNIQUE 


Following  is  an  illustrative  application  of  Gagne's 

instructional  sequencing  technique  as  described  by  Briggs  and 
Wager  (1981).  One  of  the  17  major  topical  areas  in  the  Navy 

Enlisted  Recruiting  course,  Sales  Techniques,  will  serve  as  the 
content  to  be  structured.  This  module  contains  eight  objectives 
which  are: 

1.  Perform  a simulated  sales,  interview  in  an  office  situation 
achieving  a minimum  of  75%  accuracy  based  on  the  standard 
sales  interview  checklist. 

2.  Discuss  aspects  of  generating  and  evaluating  pre-approach 
blueprinting  techniques  and  information. 

3.  Initiate  and  maintain  conversation  with  an  individual,  and 
establish  the  individual's  Dominant  Buying  Motive  (DBM). 

4.  Arouse  the  individual's  interest  and  curiosity  based  on 
his/her  DBM. 

5.  Present  the  individual  with  specific  Units  of  Conviction 
based  on  his/her  DBM. 

6.  Demonstrate  the  procedures  for  handling  the  individual's 
objections  during  the  interview. 

7.  Create  desire  in  the  individual  by  verbally  painting  a mental 
picture  based  on  the  five  natural  senses. 

8.  Close  the  sale  using  either  the  Weighing  Close  or  Minor 
Decision  Close. 

The  first  step  in  the  procedure  is  to  identify  competencies 
subordinate  to  each  course  objective.  This  procedure  is 
accomplished  by  looking  at  a particular  objective  and  asking  "What 
would  the  learner  have  to  be  able  to  do  or  know  before  being  able 
to  perform  this  objective,  given  only  instructions  as  to  what  to 
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do  on  a test  over  the  objective?"  (Briggs  and  Wager,  1981,  p.  90). 
After  two  or  three  general  skill  or  knowledge  components  are  iden- 
tified for  each  objective,  "these  [components]  are  then  examined, 
asking  the  same  question,  and  each  is  reduced  to  increasingly 
simple  skills  and  knowledges  . . . [until  the  final  identified 
component]  can  be  performed  as  a single  act. "(p. 90) 

Via  this  procedure,  those  competencies  identified  as  being 
subordinate  to  Objective  #1  were  Objectives  #2  through  #8.  The 
competency  identified  by  subject  matter  experts  as  being 
subordinate  to  Objectives  #2  through  if 8 are  indicated  in  Table  9. 

The  next  step  in  the  procedure  was  to  determine  the 
structural  interrelationships  among  all  objectives  and 
competencies.  Since  there  were  a total  of  28  objectives  and 
competencies,  789  (28  x 28)  pairs  of  relationships  had  to  be 
considered.  For  the  purpose  of  this  illustration,  consideration 
of  this  many  relationships  seemed  excessive.  In  the  interest  of 
conciseness,  the  set  of  28  elements  was  reduced  to  only  the 
original  eight  objectives  on  the  Sales  Module.  The  excluded 
comptencies  were  implicitly  subsumed  under  their  respective 
objectives.  This  action  is  legitimate  as  long  as  the  subject 
matter  specialists  recognize  the  presence  of  the  appropriate 
competencies  while  they  are  considering  the  relationship  between  a 
pair  of  objectives.  To  this  end.  Table  9 was  distributed  to  all 
subject  matter  experts  during  the  hierarchy  generation  session. 

Using  the  list  of  objectives  identified  earlier,  each 
objective  was  selected,  and  the  following  question  was  asked 
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Table  9.  Subordinate  Competencies  for  Sales  Module  Objectives 


Competencies  under  Objective  2: 

2.1  Categories  of  blueprinting  questions. 

2.2  Sources  of  blueprinting  information. 

2.3  Recording  blueprinting  information. 

2.4  Continue  blueprinting  to  qualify  individual 
and  discover  his/her  wants  & needs. 


Competencies  under  Objective  3: 

3.1  Establish  rapport. 

3.2  Question  & summarize  to  establish 
Dominant  Buying  Motive  (DBM). 

Competencies  under  Objective  4: 

4.1  Use  verbal  bridge. 

4.2  State  DBM  claim. 

4.3  Identify  product. 


Competencies  under  Objective  5/6: 


5. 1/6.1 
5. 2/6. 2 
5. 3/6.3 
5. 4/6. 4 
5. 5/6. 5 


Use  verbal  bridges. 

Present  Units  of  Conviction. 

Provide  evidence . 

Watch  and  listen  for  subtle  signals. 
Trial  close. 


Competencies  under  Objective  7: 


7.1  Remind  prospect  of  problem. 

7.2  Get  prospect's  confirmation. 

7.3  Reinforce  prospect's  solution. 

7.4  Project  prospect  to  End  Result  using  concrete  language. 
Competencies  under  Objective  8: 


8.1  Weighing  Close  for  procrastinator. 

8.2  Close  with  minor  decision. 
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relative  to  the  other  seven  objectives  in  the  set:  "Would  the 
learner  have  to  know  or  do  objective  Y before  being  able  to 
perform  objective  X?"  Via  this  inquiry  technique,  the  responses 
shown  in  Table  10  were  generated.  The  inquiry  process  required 
approximately  one  anf  one-half  hours  to  complete. 

To  facilitate  determination  of  the  hierarchical  structure 
of  these  objectives,  according  to  Briggs  and  Wager  (1981,  p.90), 
it  is  important  to  identify  the  category  of  each  objective  - 
problem  solving  skills,  rules,  concepts  or  discrimination  skills. 
These  categories  relate  to  the  Sales  Module  objectives  as  listed 
in  Table  11.  Briggs  and  Wager  (1981,  p.90)  have  stated  that 
"complete  and  empirically  tested  hierarchies  for  intellectual 
skills  objectives  appear  to  illustrate  (1)  problem  solving  skills 
require  the  use  of  two  or  more  rules;  (2)  rule-using  skills 
require  the  use  of  two  or  more  concepts;  and  (3)  concept  behaviors 
require  two  or  more  discriminations."  Using  these  guidelines, the 
relationships  in  Tables  10  and  the  classification  of  objectives  in 
Table  11,  the  hierarchy  of  Figure  10  was  produced. 

Several  departures  from  Briggs'  and  Wager's  (1981)  rules  are 
apparent  in  the  figure  (indicated  by  asterisks  in  the  figure). 
These  represent  inverted  connections  (e.g.  rules  and  concepts  lead 
to  discriminations  rather  than  vice  versa).  There  is  no  apparent 
explanation  for  these  departures.  The  solution  to  them,  though, 
is  apparent.  The  entire  process  is  repeated  until  the  resultant 
hierarchy  is  in  a form  acceptable  to  the  subject  matter 
specialists,  and  as  compliant  with  the  construction  rules  as 


possible . 
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Table  10.  Interrelationships  Among  Sales  Module  Objectives 
As  Perceived  by  Resident  Subject  Matter  Specialists 


RELATIONSHIP 

RESPONSE 

RELATIONSHIP 

RESPONSE 

1 -> 

2 

NO 

5 -> 

1 

YES 

1 -> 

3 

NO 

5 -> 

2 

NO 

1 -> 

4 

NO 

5 -> 

3 

NO 

1 -> 

5 

NO 

5 -> 

4 

NO 

1 -> 

6 

NO 

5 -> 

6 

YES 

1 -> 

7 

NO 

5 -> 

7 

NO 

1 -> 

8 

NO 

5 -> 

8 

NO 

2 -> 

1 

YES 

6 -> 

1 

YES 

2 -> 

3 

YES 

6 -> 

2 

NO 

2 -> 

4 

NO 

6 -> 

3 

NO 

2 -> 

5 

NO 

6 -> 

4 

NO 

2 -> 

6 

NO 

6 -> 

5 

YES 

2 -> 

7 

NO 

6 -> 

7 

NO 

2 -> 

8 

NO 

6 -> 

8 

NO 

3 -> 

1 

YES 

7 -> 

1 

YES 

3 -> 

2 

NO 

7 -> 

2 

NO 

3 -> 

4 

YES 

7 -> 

3 

NO 

3 -> 

5 

YES 

7 -> 

4 

NO 

3 -> 

6 

YES 

7 -> 

5 

NO 

3 -> 

7 

YES 

7 -> 

6 

YES 

3 -> 

8 

YES 

7 -> 

8 

YES 

4 -> 

1 

YES 

8 -> 

1 

YES 

4 -> 

2 

NO 

8 -> 

2 

NO 

4 -> 

3 

NO 

8 -> 

3 

NO 

4 -> 

5 

YES 

8 -> 

4 

NO 

4 -> 

6 

NO 

8 -> 

5 

YES 

4 -> 

7 

YES 

8 -> 

6 

YES 

4 -> 

8 

YES 

8 -> 

7 

YES 

Note:  For  ease  in  reading  this  table,  the  verbiage  of  the 

questions  is  symbolically  represented  as  follows: 

X ->  Y = "Would  the  learner  have  to  know  or  perform 

objective  X before  accomplishing  objective  Y?" 
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Table  11.  Classification  of  Sales  Module  Objectives 


Objective 

Number 

1 

2 

3 

4 

5 

6 
7 


Classification 
Problem  Solving 
Concept 
Concept 
Rule 
Rule 
Concept 
Rule 


8 


Discrimination 
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2J0TE:  Asterisks  on  certain  vectors  in  the  hierarchy  indicate 

departures  from  the  interconnection  rules  established  by 
Gagne,  as  enumerated  by  Briggs  and  Wager  (1981). 


Figure  10.  A Learning  Hierarchy  for  the  Sales  Module 
of  the  Navy  Enlisted  Recruiter  Course 
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The  process  was  repeated  with  the  subject  matter 

specialists.  In  reviewing  their  initial  responses  (cf.  Table  10), 
they  indicated  unanimously  that  there  were  no  changes  to  be  made. 
The  next  step,  then,  was  to  examine  the  hierarchy  and  adjust  it  to 
reduce  as  many  erroneous  connections  as  possible.  (Note:  these 

erroneous  connections  are  indicated  in  Figure  10  with  asterisks.) 
In  making  this  adjustment,  all  bi-directionally  connected 
objectives  were  placed  on  the  same  level  of  the  hierarchy.  The 
adjustments  were  examined  by  the  subject  matter  specialists  after 
they  were  accomplished  to  determine  if  any  discrepancies 
remained.  The  specialists  indicated  after  this  adjustment  process 
that  the  hierarchy  appeared  to  be  more  closely  represent  the 
theoretical  interconnection  of  content  than  in  the  previous 
hierarchy.  Thus,  the  process  was  terminated.  The  resulting 
hierarchy  is  shown  in  Figure  11.  The  entire  process  from 
beginning  to  end  took  approximately  3 hours.  This  hierarchy  is 
extremely  similar  in  form  to  that  developed  with  the  aid  of  the 
Interpretive  Structural  Modeling  methodology  (cf.  Figure  14, 
Appendix  B). 

Although  it  is  not  evident  from  this  simple  illustration, 
such  a trial— and-error  process  is  both  mentally  taxing  and 
time-consuming.  The  labor-intensiveness  of  this  hierarchy 
construction  process  is  in  direct  proportion  to  the  number  of 
objectives  under  consideration. 
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1 


Figure  11. 


An  Adjusted  Gagne-type  Hierarchy  for  the  Sales  Module 


Appendix  B 


AN  APPLICATION  OF  INTERPRETIVE  STRUCTURAL  MODELING  TO 
INSTRUCTIONAL  SEQUENCING 


This  appendix  illustrates  how  the  Interpretive  Structural 
Modeling  (ISM)  methodology  was  applied  to  the  task  of  constructing 
an  instructional  hierarchy.  In  order  to  compare  the  ISM 
methodology  with  Gagne's  procedure,  the  Sales  Module  of  the  Navy 
Enlisted  Recruiter  Course,  the  same  course  segment  used  in 
Appendix  1,  will  also  be  used  here. 

Renckly's  (1981)  five-step  process  for  applying  ISM  to  the 
hierarchy  construction  problem  is  restated  below: 

1.  Identify  objectives  of  the  curriculum. 

2.  Determine  a transitive  relation  that 
applies  to  the  objectives  in  the  context 
of  the  instructional  situation. 

3.  Place  the  objective  relationships  into  a 
matrix  format  . . . 

4.  Manipulate  the  matrix  into  a suitable 
form  - called  a reachability  matrix. 

5.  Re-order  the  rows  and  columns  of  the 
reachability  matrix  and  partition  it  to 
reflect  hierarchical  levels  . . . p.  203 

This  procedure  was  programmed  into  an  APPLE  II  Plus  microcomputer 

Although  the  manipulative  process  performed  by  the  program  is 

ordinarily  transparent  to  the  user,  it  will  be  explained  in 

detail  for  the  sake  of  illustration. 

Step  1 

The  objectives  identified  for  hierarchical  ordering  are 
identical  to  those  stated  in  Appendix  A.  They  are  enumerated 
again  as  follows: 
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1.  Perform  a simulated  sales  Interview  in  an  office  situation 
achieving  a minimum  of  75%  accuracy  based  on  the  standard 
sales  interview  checklist. 

2.  Discuss  aspects  of  generating  and  evaluating  pre- approach 
blueprinting  techniques  and  information. 

3.  Initiate  and  maintain  conversation  with  an  individual,  and 
establish  the  individual's  Dominant  Buying  Motive  (DBM). 

4.  Arouse  the  individual's  interest  and  curiosity  based  on 
his/her  DBM. 

5.  Present  the  individual  with  specific  Units  of  Conviction 
based  on  his/her  DBM. 

6.  Demonstrate  the  procedures  for  handling  the  individual's 
objections  during  the  interview. 

7.  Create  desire  in  the  individual  by  verbally  painting  a mental 
picture  based  on  the  five  natural  senses . 

8.  Close  the  sale  using  either  the  Weighing  Close  or  Minor 
Decision  Close. 

Step  2 


The  transitive  relation  is  necessary  to  comprehend  or 
accomplish  was  agreed  upon  by  the  subject  matter  specialists 
involved  in  this  hierarchy  generation  process.  This  relation  was 
comparable  in  meaning  to  the  question  used  to  structure  the  Sales 
Module  objectives  via  the  Gagne  procedure  (cf.  Appendix  1).  This 
relation  was  combined  with  each  pair  of  objectives  to  form  a 
question,  which  is  generically  stated  as  follows: 


"Is  Objective  X necessary  to  comprehend 
or  accomplish  Objective  Y?" 
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Step  3 

Since  there  were  eight  objectives  in  the  set,  the  subject 

matter  specialists  provided  answers  for  64  (8  x 8)  distinct 

questions.  For  each  question  answered  yes,  the  computer  placed  a 
1 into  the  corresponding  X-Y  cell  of  a 8x8  matrix.  Likewise,  for 
each  question  answered  no,  the  computer  placed  a 0 into  the 

corresponding  cell  of  the  matrix.  The  completed  matrix  required 
approximately  30  minutes  to  construct,  and  is  illustrated  in 

Figure  12(a). 

The  matrix  of  Figure  12(a)  is  known  as  an  adjacency  matrix 
because  it  contains  information  (Is)  on  those  objectives  that  are 
adjacent  (i.e.  directly  connected)  to  one  another  in  the 
hierarchical  structure.  However,  there  is  insufficient 

information  in  this  matrix  to  determine  the  form  of  the  hierarchy 
(i.e.  the  number  of  levels  in  the  hierarchy,  the  objectives  within 
each  level,  etc.). 

By  definition,  a Warfield  (1976)  defines  a hierarchy  as  a 
multi-level  structure,  while  Gagne  (1977)  describes  such  a 
structure  as  being  composed  of  pair-wise  relationships  between 
objectives  which  are  either  dependent,  supportive  or  independent. 
If  an  objective  is  determined  to  be  dependent  in  relation  to 
another  objective,  the  two  must  necessarily  lie  on  different 
levels  of  the  hierarchy.  Similarly,  a pair  of  objectives  bound  in 
a mutually  supportive  relationship  must,  by  necessity,  lie  on  the 
same  level  of  the  hierarchy.  Finally,  objectives  which  are 
independent  of  all  others  in  the  set  may  or  may  not  lie  on  the 
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1 2 3 4 5 6 7 8 
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(a) 


(b) 


(c) 


(d) 


Figure  12.  Three  Iterations  in  the  Matrix  Multiplication  Process 
(a)  the  original  adjacency  matrix,  (b)  the  first 
iteration,  (c)  the  second  iteration  and  (d)  the  final 
reachability  matrix  for  the  Sales  Module  objectives. 
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same  level  as  some  other  objective,  but,  in  either  event,  the 
independent  objective  is  not  interconnected  with  any  other  in  the 
hierarchy. 

Step  4 

In  this  step,  the  adjacency  matrix  is  multiplied  by  itself 
(raised  to  powers)  via  binary  multiplication.  Each  multiplication 
will  change  some  of  the  originally  supplied  Os  in  the  matrix  to 
Is.  Originally  supplied  Is  are  not  affected.  Eventually,  the 

multiplication  process  will  reach  a convergence  point,  wherein 
further  multiplications  will  no  longer  change  Os  to  Is.  When 
convergence  is  achieved,  the  resulting  matrix  is  referred  to  as 
the  reachability  matrix.  This  matrix  contains  information  on  the 
interconnections  between  objectives  (from  the  adjacency  matrix), 
as  well  as  information  detailing  the  number  and  composition  of  the 
levels  within  the  hierarchy.  Warfield  (1976)  has  shown  that  the 
convergence  point  for  any  adjacency  matrix  can  be  achieved  in  N or 
fewer  multiplications,  where  N is  the  number  of  objectives  in  the 
set.  For  the  current  illustration,  the  reachability  matrix  was 
obtained  after  three  iterations.  These  three  iterations  are  shown 
in  Figure  12(b),  (c)  and  (d). 

Step  5 

To  extract  information  concerning  the  form  of  the  hierarchy 
from  the  reachability  matrix,  the  matrix  is  modified  by  row  and 
column  interchanges.  This  is  efficiently  accomplished  by 
constructing  partition  tables.  Table  12  provides  an  example  of  a 

such  a partition  table  constructed  from  the  reachability  matrix  of 
Figure  12(d). 
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Table  12.  A Partition  Table  for  the  Matrix  of  Figure  12(d) 


Obj  ective 
Numbe  r 

Reachability 

Set 

Antecedent 

Set 

Intersection 

Set 

1 

1 

1,  2,  3,  4,  5, 
6,  7,  8 

1 

2 

1,  2,  3,  4,  5,  6, 
7,  8 

2 

2 

3 

1,  3,  4,  5,  6,  7, 
8 

2,  3 

3 

4 

1,  4,  5,  6,  7,  8 

2,  3,  4,  5,  6, 
7,  8 

4,  5,  6,  7,  8 

5 

1 , 4,  5,  6,  7,  8 

2,  3,  4,  5,  6, 
7,  8 

4,  5,  6,  7,  8 

6 

1,  4,  5,  6,  7,  8 

2,  3,  4,  5,  6, 
7,  8 

4,  5,  6,  7,  8 

7 

1,  4,  5,  6,  7,  8 

2,  3,  4,  5,  6, 
7,  8 

4,  5,  6,  7,  8 

8 

1,  4,  5,  6,  7,  8 

2,  3,  4,  5,  6, 
7,8 

4,  5,  6,  7,  8 
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The  table  consists  of  three  columns:  a reachability  set,  an 
antecedent  set  and  an  intersection  set.  The  reachability  set  for 
objective  #1  is  constructed  by  inspecting  row  #1  of  the 
reachability  matrix  (Figure  12c).  Every  1 in  that  row  corresponds 


to  a column 

index. 

and  every 

such 

column  index  is  in 

the 

reachability 

set  of 

objective 

#1. 

The  antecedent  set 

for 

objective  #1  is  found  by  inspecting  column  #1  of  the  matrix. 
Every  1 in  column  #1  corresponds  to  a row  index,  and  the  set  of 
such  row  indices  comprises  the  antecedent  set  for  objective  #1. 
The  intersection  set  of  objective  #1  is  determined  by  finding  the 
Boolean  intersection  of  its  reachability  set  and  the  antecedent 
sets.  The  intersection  set  contains  all  elements  that  simultan- 
eously reside  in  both  the  reachability  set  and  the  antecedent 
set.  In  Figure  12(d),  the  row  and  column  indices  (1-8)  identify 
the  respective  elements  of  the  reachability,  antecedent  and 
intersection  sets. 

Once  constructed,  the  table  is  examined  to  identify  those 
objectives  whose  reachability  set  equals  its  intersection  set. 
Such  objectives  are  defined  as  members  of  the  same  level  (i.e. 
they  are  mutually  supportive)  in  the  hierarchy.  The  reason  for 
this  is  because  every  member  of  the  intersection  set  must  simul- 
taneously be  antecedent  to,  and  reachable  from,  a given  objec- 
tive. For  this  to  be  true,  the  objective  must  lie  on  the  same 
level  as  all  others  which  meet  these  same  conditions.  Consider 
the  following  illustration: 
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Let  <R>  be  a relation  which  binds  a set  of  objectives  (say  A, 
B and  C) . Assume  that  objective  B is  in  the  reachability  set  of 
objective  A (i.e.,  A <R>  B).  Assume,  also,  that  objectives  B and 
C are  both  members  of  the  antecedent  set  of  objective  A (i.e., 
B <R>  A and  C <R>  A).  The  intersection  of  these  two  sets  is 
objective  B,  which  satisfies  the  following  relationships 
simultaneously: 

A <R>  B 

B <R>  A 

Under  these  conditions,  objectives  A and  B are  symmetrically 
related.  That  is,  they  are  in  a supportive  relationship  with  each 
other.  Thus,  they  must  both  occupy  the  same  level  in  the 
hierarchy.  This  argument  easily  extends  to  any  number  of 
objectives  which  meet  the  same  conditions.  From  the  table,  it  is 
clear  that  only  objective  #1  meets  this  condition.  It,  therefore, 
occupies  the  uppermost  level  in  the  hierarchy. 

The  top  level  of  the  hierarchy  is  now  identified.  All 
references  to  objective  #1  in  the  table  are  removed,  and  a reduced 
(second-level)  table  is  constructed  (see  Table  13).  Again,  the 
table  is  examined  to  identify  those  objectives  for  which  the 
reachability  and  intersection  sets  match.  Once  found,  these 
objectives  compose  the  second  level  of  the  hierarchy.  In  this 
case,  the  second  level  is  composed  of  objectives  4,  5,  6,  7 and 
8.  Again,  all  references  to  these  objectives  are  removed  from  the 
table,  and  another  reduced  (third-level)  table  is  constructed,  as 
shown  in  Table  14.  From  this  table,  objective  #3  is  identified  as 


Table  13.  A Reduced  Partition  Table  - Level  1 Removed 


Objective 

Number 

Reachability 

Set 

Antecedent 

Set 

Intersection 

Set 

2 

2,  3,  4,  5,  6, 
7,  8 

2 

2 

3 

3,  4,  5,  6,  7, 
8 

2,  3 

3 

4 

4,  5,  6,  7,  8 

2,  3,  4,  5,  6, 
7,  8 

4,  5,  6,  7,  8 

5 

4,  5,  6,  7,  8 

2,  3,  4,  5,  6, 
7,  8 

4,  5,  6,  7,  8 

6 

4,  5,  6,  7,  8 

2,  3,  4,  5,  6, 
7,  8 

4,  5,  6,  7,  8 

7 

4,  5,  6,  7,  8 

2,  3,  4,  5,  6, 
7,  8 

4,  5,  6,  7,  8 

8 

4,  5,  6,  7,  8 

2,  3,  4,  5,  6, 
7,  8 

4,  5,  6,  7,  8 
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Table  14.  A Reduced  Partition  Table  - Levels  1 and  2 Removed 


Objective 

Reachability 

Antecedent 

Intersection 

Number 

Set 

Set 

Set 

2 

2,  3 

2 

2 

3 

3 

2,  3 

3 
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occupying  the  third  level  of  the  hierarchy.  Removing  this 
objective  from  the  table  yields  a fourth-level  partition  table. 
However,  at  this  point,  only  objective  #2  remains.  Therefore,  it 
is  designated  as  occupying  the  fourth  level  of  the  hierarchy. 


When 

the 

partition 

table 

is  voided  of 

all  objectives. 

the 

partitioning 

process  and  the 

hierarchy  are  complete. 

The 

resultant 

ordered  set  is 

: (1; 

4,  5,  6,  7, 

8;  3;  2).  This 

set 

provides 

the 

row  and 

column 

interchange 

sequence  for 

the 

reachability  matrix.  The  modified  matrix  is  shown  in  Figure  13. 
This  matrix  is  useful  in  analyzing  complex  subsets  of  supporting 
objectives  which  are  interconnected  on  the  same  level  of  the 
hierarchy,  where  three  or  more  objectives  are  mutually 
interconnected . 

The  resultant  hierarchy  is  constructed  in  three  levels,  each 
of  which  contain  the  objectives  indicated  in  Figure  13.  To 
determine  the  precise  interconnections  among  objectives  in  the 
hierarchy,  the  adjacency  matrix  is  consulted.  The  Is  in  the 
adjacency  matrix  of  Figure  12(a)  indicate  the  interconnections. 
For  example,  a 1 in  the  (5,6)  cell  of  the  adjacency  matrix 
indicates  that  objective  5 is  connected  to  objective  6 (i.e.  the 
arrowhead  of  the  interconnecting  vector  points  to  objective  6). 
The  completed  hierarchy  for  the  objectives  of  the  Sales  Module  is 
displayed  in  Figure  14. 


Once 

the  hierarchy  is 

produced,  the 

subject  matter 

specialists 

review  it,  asking 

the 

question: 

"In  reality, 

is 

objective 

X connected  to  objective 

Y in  the 

same  manner 

as 

109 


1 

4 

5 

6 

7 

8 

3 

2 

1 

1 

0 

0 

0 

0 

0 

0 

0 

4 

1 

1 

1 

1 

1 

1 

0 

0 

5 
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1 

1 

1 

1 

1 

0 

0 

6 

1 

1 

1 

1 

1 

1 

0 

0 

7 

1 

1 

1 

1 

1 

1 

0 

0 

8 

1 

1 

1 

1 

1 

1 

0 

0 

3 

1 

1 

1 

1 

1 

1 

1 

0 

2 

1 

1 

1 

1 

1 

1 

1 

1 

- Level  1 
^ e Level  1 

- Level  3 

- Level  4 


Figure  13.  A Modified  Reachability  Matrix  Containing  Four  Levels 
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1 


Figure  14.  An  ISM-Developed  Hierarchical  Structure 
for  the  Sales  Module  of  the  Enlisted  Recruiter  Course 


Ill 


depicted  on  the  hierarchy?"  The  possibility  exists  that  such  an 
inquiry  will  reveal  the  hierarchy  to  be  in  error.  If  this  is  the 
case,  Renckly ' s five-step  procedure  is  reiterated.  The  primary 
advantage  of  the  ISM  technique  over  Gagne's  methodology  is  that 
the  subject  matter  specialists  need  only  to  re-manipulate  those 
originally  supplied  data  elements  which  were  found  to  be  in  error 
on  the  hierarchy  in  order  to  recompute  a revised  hierarchy.  The 
ISM  algorithms  handle  the  inferential  ramifications  of  the  subject 
matter  specialists'  changes.  In  Gagne's  method,  the  subject 
matter  specialists  must  explicitly  reconsider  most  of  the 
originally  supplied  data  elements  to  determine  the  full  extent  of 
the  intended  hierarchy  changes.  The  advantage  offered  by  ISM 
alleviates  the  subject  matter  specialists  from  the  necessity  of 
investing  an  excessive  amount  of  mental  effort  in  protracted, 
repetitive  hierarchy  revision  sessions.  Thus,  the  labor-intensive 
nature  of  Gagne's  method,  which  becomes  clearly  evident  during  the 
hierarchy  revision  process,  is  overcome  by  the  use  of  ISM's 
inferential  algorithmic  procedure,  and  is  made  even  more  useful  by 


the  intervention  of  the  computer. 
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POST  GRADUATE  SURVEY  QUESTIONNAIRE 


The  following  questions  were  used  to  gather  information 

concerning  course  graduates'  perceptions  of  how  applicable  their 

school  training  was  to  real-world  recruiting  practices. 

1.  How  well  did  you  understand  the  operation  of  the  CNRC  Chain 
of  Command  ? 

2.  How  well  did  you  understand  what  you  must  do  to  become 
qualified  in  accordance  with  RQS? 

3.  How  well  did  you  understand  RPPMS  (Freeman  Plan)? 

4.  How  well  could  you  blueprint? 

5.  How  well  could  you  establish  rapport? 

6.  How  well  could  you  arouse  curiosity? 

7.  How  well  could  you  provide  conviction? 

8.  How  well  could  you  create  desire? 

9.  How  well  could  you  close? 

10.  How  well  could  you  handle  objections? 

11.  How  well  could  you  deliver  speeches? 

12.  How  well  could  you  make  appointments  using  the  telephone? 

13.  How  well  could  you  obtain  referral  leads  and  make 
appointments  with  them? 

14.  How  well  could  you  handle  responses? 

15.  How  well  could  you  use  mass /personal  mailouts? 

16.  How  well  could  you  personally  develop  contacts 

(face-to-face)? 

17.  How  well  could  you  complete  a standard  pre-enlistment  kit? 

18.  How  well  could  you  complete  a kit  requiring  a waiver? 

19.  How  well  could  you  complete  a kit  for  a prior-service 

applicant? 
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20.  How  well  could  you  complete  a TAB  F? 

21.  How  well  could  you  complete  a TAB  G? 

22.  How  well  could  you  complete  a TAB  H? 

23.  How  well  could  you  analyze  a TAB  A? 

24.  How  well  could  you  fill  out  a prospect  card? 

25.  How  well  did  you  understand  the  purpose  of  the  Prospect 

Card  System? 

26.  How  well  did  you  understand  the  use  of  the  Prospect  files? 

27.  How  well  did  you  understand  the  use  of  the  SMART  System? 

28.  How  well  could  you  develop  an  itinerary  from  displayed  SMART 
information? 

29.  How  well  did  you  understand  the  proper  use  and  security 

requirements  of  the  credit  card  along  with  it's  limits? 

30.  How  well  did  you  understand  the  procedures  for  emergency 
vehicle  repairs? 

31.  How  well  did  you  understand  CNRC  domicile-to-duty  policy? 

32.  How  well  did  you  understand  vehicle  safety  requirements? 

33.  How  well  did  you  understand  the  function  of  the  Education 
Specialist? 

34.  How  well  did  you  understand  the  function  of  the  Recruiting 
District  Assistance  Council? 

35.  How  well  did  you  understand  the  function  of  the  Campus 

Liaison  Officer? 

36.  How  well  did  you  understand  the  function  of  the  Youth 

Programs  Officer/Petty  Officer? 
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RECRUITER  CANVASSER  JOB  TASK.  INVENTORY 


The  following  task  inventory,  excerpted  from  the  United  States 
Navy  Recruiting  Command  Enlisted  Recruiter  Canvasser  Course: 
Program  of  Instruction  (1984),  lists  the  elements  comprising  the 
job  of  the  Enlisted  Navy  Recruiter.  Items  below  without  asterisks 
are  designated  as  necessary  to  be  trained  at  the  resident  school. 
Those  with  asterisks  are  NOT  trained  at  the  school. 


1.0  ADMINISTRATION 


1.1  Continuously  employ  all  required  security  measures. 

1.1.1  Identify  security  requirements  for  NRS. 

1.1.2  Identify  security  requirements  for  test 
materials . 

1.1.3  Identify  security  requirements  for  Government 
TRs,  transportation  and  meal  tickets. 

1.1.4  Identify  prospect  information  security 
requirements . 

1.1. 4.1  Privacy  Act 

1.1. 4. 2 Freedom  of  Information  Act 

1.1.5  Apply  security  measures. 


* 1.2 


Compile  station  logs  and  submit  reports. 


* 

* 


* 

* 

* 


1.2.1  Identify  pertinent  station  logs/reports. 

1.2.2  Explain  the  purpose  of  logs  and  reports. 

* 1.2. 2.1  Motor  vehicle  reports 

* 1.2. 2. 2 Bulk  ticket  logs 

* 1.2. 2. 3 Out-of-pocket  expense  sheet 

* 1.2. 2. 4 Motor  vehicle  logs 

* 1.2. 2. 5 Individual  evaluation  input  sheet 

1.2.3  Research  procedures  for  compiling  station 
logs  and  reports. 

1.2.4  Continuously  maintain  delegated  station  logs. 

1.2.5  Submit  reports  as  required. 


1.3  Use  CNRC  production  chain  of  command. 

1.3.1  Identify  the  CNRC  production  chain  of 

command . 

* 1.3.2  Identify  incumbents  by  name  in  the  CNRC 
production  chain  of  command. 
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* 1.4 

* 1.5 


1.3.3  Explain 

1.3. 3.1 

1.3. 3. 2 

1.3. 3. 3 

1.3. 3. 4 


the  use  of  the  chain  of  command. 
Information  flow 
Individual  requests 
Point  papers 
Order  passing 


Follow  directives  of  higher  authority. 


Become  qualified  in  accordance  with  Recruiter 
Qualification  Standards  (RQS). 

1.5.1  Describe  the  purpose  and  minimum  standards 
for  qualification  under  RQS. 

1.5.2  Describe  the  operation  of  the  Recruiter 
Personnel  Production  Management  System. 


2.0  MARKETING 


2.1 


Identify  target  market  populations. 


2.1.1  Identify  assigned  territory  boundaries. 

2.1.2  Identify  educational  institutions. 

2. 1.2.1  High  schools 

2. 1.2. 2 Colleges 

2. 1.2. 3 Community  Colleges 

2. 1.2.4  Vocational/Technical  schools 

2. 1.2. 5 Quality  of  educational  institutions 

2.1.3  Identify  population  centers. 

2. 1.3.1  Metropolitan  areas 

2. 1.3. 2 Cities 

2 . 1 . 3 . 3 Towns 

2. 1.3.4  Unincorporated  areas 

2.1.4  Identify  leisure-time  activity  centers. 

2. 1.4.1  Youth  centers 

2. 1.4. 2 Pool  halls 

2. 1.4. 3 Athletic  leagues 

2. 1.4. 4 Restaurants 

2. 1.4. 5 Automotive  centers 

2.1.5  Provide  information  on  assigned  territory. 


2.2 


Use  Navy  Recruiting  District  (NRD)  resources 


2.2.1  Identify  recruiter  support  functions  of  each 
department . 

2. 2. 1.1  Officer  Prograns 

2. 2. 1.2  Enlisted  Programs 

2. 2. 1.3  Processing 

2. 2. 1.4  Classification 

2.2. 1.5  Recruiting  Aids 

2. 2. 1.6  Public  Affairs 

2. 2. 1.7  Supply 

2. 2. 1.8  Administrative 
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2.2. 1.9  Education  Specialist 

2.2.1.10  Lead  Tracking  Center 


2.2.2  Use  and  maintain  allocated  equipment. 

* 2. 2.2.1  Identify  equipment. 

* 2.2.2. 1.1  Fairchild  projector 

* 2. 2. 2. 1.2  35MM  Slide  projector 

* 2. 2. 2. 1.3  16MM  Motion  picture 

projector 

* 2. 2. 2. 1.4  Codaphone 

* 2. 2. 2. 1.5  Typewriter 

* 2.2.2. 1.6  Polaroid  camera 

2. 2.2. 2 Operate  equipment. 

2. 2. 2. 3 Conduct  routine  maintenance. 

2. 2. 2. 4 Perform  minor  repairs. 

2. 2. 2. 5 Describe  equipment  replacement 
procedures . 


* 

* 

•k 

k 


2.3  Use  community  personnel  resources  to  contribute  to 

the  recruiting  effort. 


2.3.1 


2.3.2 

2.3.3 


2.3.4 

2.3.5 


Identify  community  personnel  who  could 
assist  in  the  recruiting  effort. 

2. 3. 1.1  Local  education  personnel 

2. 3. 1.2  Labor  personnel 

2. 3. 1.3  Government  personnel 

2. 3. 1.4  Reserve  activities  personnel 

2. 3. 1.5  Recruiting  Assistance  Program 
personnel 

2.3. 1.6  Hometown  Assistance  Recruiting 
Program  personnel 

2. 3. 1.7  Delay  Entry  Program  personnel 

2. 3. 1.8  Other  Navy  enlistees 

2. 3. 1.9  Military  retired  personnel 

2.3.1.10  Naval  Academy  Liaison  personnel 

2.3.1.11  Recruiting  District  Assistance 
Council  personnel 

2.3.1.12  Local  media  personnel 

2.3.1.13  Civic  organizations 

2.3.1.14  Youth  organizations 

2.3.1.15  Professional  organizations 

2.3.1.16  Veterans'  organizations 

2.3.1.17  Veterans'  counselors 

2.3.1.18  Parents  of  enlistees 

2.3.1.19  Minority  organizations 

2.3.1.20  Local  business  owners /managers 
Identify  how  they  may  assist. 

Convince  community  personnel  to  contribute 
to  the  recruiting  effort  . 

2. 3. 3.1  Use  sales  skills. 

Request  specific  assistance  and  follow-up. 
Recognize  effective  assistance. 
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3.0  PROSPECTING 

3.1  Operate  Government  vehicles  in  accordance  with 

current  directives. 


* 


* 


3.1.1  Possess  a Government  driver's  license. 

3.1.2  Successfully  complete  a defensive  driving 
course . 

3.1.3  Describe  vehicle  operating  instructions. 

3.1.4  Describe  all  safety  instructions. 

3.1.5  Describe  instructions  for  use  of  credit 
cards . 

3. 1.5.1  Fuel  procurement 

3. 1.5. 2 Minor  repairs 

3. 1.5.3  Limitations  on  use 

3. 1.5. 4 Replacement  procedures 

3.1.6  Successfully  complete  a road  test. 


3.2  Use  the  prospect  card  system. 

3.2.1  Obtain  prospect  names. 

3. 2. 1.1  School  lists 

3.2. 1.2  High  school  yearbook 

3. 2. 1.3  Newspaper  articles 

3. 2. 1.4  Athletic  rosters 

3. 2. 1.5  Purchased  lists 

3.2.2  Transcribe  all  prospect  names  to  prospect 
cards . 

3.2.3  Describe  the  prospect  card  filing  system. 

3. 2. 3. 2 Working  file 

3. 2. 3. 3 School  file 

3. 2. 3. 4 College  file 

3. 2. 3. 5 Direct  file 

3. 2. 3. 6 Prior-service  file 

3.2.4  Continuously  update  prospect  information. 

3.2.5  Obtain  and  return  prospect  cards  to 
Recruiter-in-Charge . 

3.3  Make  appointments  by  developing  contacts. 


3.3.1  Make  appointments  by  telephone. 

3. 3. 1.1  Use  a telephone  technique. 

3.3. 1.2  Document  information. 

3. 3. 1.3  Record  appointment  on  calendar. 

3. 3. 1.4  Document  activity. 

3.3.2  Make  appointments  via  personal  contacts. 

3.3.2. 1 Initiate  face-to-face  contact. 

3. 3. 2. 2 Ask  for  appointment. 

3. 3. 2. 3 Document  information. 

3. 3. 2. 4 Record  appointment  on  calendar. 

3. 3. 2. 5 Document  activity. 
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3.3.3  Make  appointments  with  referrals. 

3. 3. 3.1  Ask  for  referrals. 

Initiate  contact. 

Ask  for  appointment. 

Document  information. 

Record  appointment  on  calendar. 
Document  activity. 

3.3.4  Use  mail-outs. 

3.3.4. 1 Coordinate  mass  mail-outs. 

Use  personalized  mail-outs. 
Handle  responses. 

Document  information. 

Document  activity. 


3. 3. 3. 2 

3. 3. 3. 3 

3. 3. 3. 4 

3. 3. 3. 5 

3. 3. 3. 6 


3. 3. 4. 2 

3. 3. 4. 3 

3. 3. 4. 4 

3. 3. 4. 5 


3.3.5  Make  appointments  with  advertising  leads. 


3.3.5. 1 

Pursue  all  advertising  leads. 

3. 3. 5. 2 

Ask  for  appointment. 

3. 3. 5. 3 

Document  information. 

3. 3. 5. 4 

Record  appointment  on  calendar 

3. 3.5. 5 

Document  activity. 

3. 3. 5. 6 

Provide  feedback  on  lead  to 
District  Lead  Tracking  Center. 

3.4  Develop  a prospecting  plan. 


3.4.1  Obtain  minimum  recruiting  goals  from 
supervisor . 

3.4.2  List  all  available  resources. 

3.4.3  List  uses  of  each  resource. 

3.4.4  Determine  required  prospecting  activity 
levels . 

3.4.4. 1 Telephone 

3. 4. 4. 2 Referrals 

3. 4. 4. 3 Personally  developed  contacts 

3. 4. 4. 4 Mail-outs 

3. 4. 4. 5 Advertising 

3.4.5  Develop  itineraries  using  the  SMART  system 
as  a resource. 

3.4.6  Use  planning  calendars. 

3.4.7  Schedule  all  activity  within  specific  time 
frames . 

3.4.8  Submit  to  supervisor  for  approval. 


* 3.5 

3.6 


Implement  approved  prospecting  plan. 
Improve  existing  prospecting  plan. 


3.6.1  Analyze  results. 

3.6.2  Analyze  activity. 

3.6.3  Adjust  activity. 

3.6.4  Draft  new  plan. 

3.6.5  Obtain  approval. 
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4.0  SELLING 


4.1 


Use  sales  skills. 


4.1.1  Generate  and  evaluate  pre-approach  blueprint 
information . 

4. 1.1.1  Categories  of  blueprinting  questions 
Sources  of  blueprinting  information 
Recording  blueprinting  information 
Continue  blueprinting  to  qualify 
and  discover  wants  and  needs  of 
prospect . 

4.1.2  Initiate  and  maintain  conversation. 

Establish  rapport. 

Question  and  summarize  to  establish 
Dominant  Buying  Motive  (DBM). 

4.1.3  Arouse  curiosity/interes t . 

4. 1.3.1  Use  verbal  bridges. 

State  DBM  claim. 

Identify  product. 

4.1.4  Provide  conviction. 

4. 1.4.1  Use  verbal  bridges. 

Present  unit  of  conviction. 

Provide  evidence. 

Watch  and  listen  for  subtle  signals. 
Trial  close 
Handle  objections. 

4.1.5  Create  desire. 

4. 1.5.1  Remind  prospect  of  problem. 

Get  prospect's  confirmation. 
Reinforce  proposed  solution. 

Project  prospect  to  end  result  using 
concrete  language . 

4.1.6  Close  the  sale. 

4. 1.6.1  Weighing  close  for  procrastinator 

4. 1.6.2  Close  with  minor  decision. 


4. 1.1.2 

4. 1.1. 3 

4. 1.1. 4 


4. 1.2.1 

4.1 .2.2 


4. 1.3. 2 

4. 1.3.3 


4. 1.4. 2 

4. 1.4.3 

4. 1.4. 4 

4. 1.4.5 

4. 1.4. 6 


4. 1.5.2 

4.1 .5.3 

4. 1.5. 4 


4.2 


Conduct  Interviews. 


4.2.1  Plan  approach. 

4.2. 1.1  Review  prospect's  documented  data. 

4.2.2  Incorporate  sales  skills  in  presentation. 

4. 2. 2.1  Tailor  presentation  to  wants  and 
needs  of  interviewee . 

4. 2. 2. 2 Practice  law  of  psychological 
reciprocity  throughout  interview. 

4. 2. 2. 3 Do  not  challenge  beliefs. 

4. 2. 2. 4 Forget  self  completely. 

4. 2. 2. 5 Maintain  control  of  conversation. 

4.2.3  Analyze  results. 
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4.3  Qualify  potential  enlistees. 

4.3.1  Basic  Enlistment  Eligibility  Requirements 

4.3.2  Waivers 

4.3.3  Prior-service 

4.4  Present  Navy  image  to  community. 

4.4.1  Deliver  speeches. 

4. 4. 1.1  Promote  awareness. 

4.4.2  Maintain  community  relations. 

4. 4. 2.1  Personal  appearance 

4. 4. 2. 2 Personal  conduct 

4. 4. 2. 3 Involvement  in  community  affairs 

4.5  Document  activity. 

4.5.1  Activity  Analysis  System 

4.5.2  Prospect  card  system 

4.5.3  National/local  Lead  Tracking  System 


5.0  PROCESSING 


5.1  Prepare  pre-enlistment  kits. 

5.1.1  Complete  a standard  pre-enlistment  kit. 

5.1.2  Complete  a pre-enlistment  kit  requiring  a 
waiver . 

5.1.3  Complete  a prior-service  pre-enlistment  kit. 

* 5.2  Provide  assistance  to  further  process  the 

applicant . 


* 5.2.1  Schedule  applicant. 

* 5.2. 1.1  MET  site 

* 5. 2. 1.2  MEPS 

* 5. 2. 1.3  NRD 

* 5.2.2  Provide  transportation. 

* 5.2.3  Brief  applicant  on  scheduled  procedures. 

5.3  Follow  guidelines  in  established  recruiting 

directives . 


5.4  Handle  returned  applicants. 

5.4.1  Indoctrinate  DEP  personnel  on 
responsibilities . 

5.4.2  Pursue  Qualified-Not-Enlisted  applicants. 

5.4.3  Counsel  permanently  rejected  applicants. 
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5.5 


5.4.3. 1 


5. 4. 3. 2 


5.4. 3.3 

5.4.4  Counsel 
5.4.4. 1 


5. 4. 4. 2 


5. 4. 4. 3 


Determine  if  applicant  harbors  ill 
will  toward  Navy  because  of 
rejection . 

Demonstrate  empathy  for  his/her 
situation . 

Attempt  to  neutralize  his/her 
negative  attitude  toward  the  Navy, 
temporarily  rejected  applicants. 
Determine  if  applicant  harbors  ill 
will  toward  Navy  because  of 
rejection. 

Demonstrate  empathy  toward  his/her 
situation . 

Inform  applicant  what  he/she  must 
do  to  become  enlistment  eligible. 


Improve  processing  results 


5.5.1  Document  activity. 

5. 5. 1.1  TAB  H 

5.5. 1.2  Prospect  cards 

5.5.2  Analyze  results. 

5.5.2. 1 Accessions 

5.5.3  Analyze  activity. 

5.5.3. 1 Interviews 

5. 5. 3. 2 ASVABs 

5. 5. 3. 3 Physicals 

5. 5. 3. 4 Screen  tests 

5.5.4  Adjust  activity. 


Appendix  E 


RECRUITER  CANVASSER  COURSE  OBJECTIVES 


The  following  list  of  course  objectives  was  excerpted  from 
the  United  States  Navy  Recruiting  Command  Enlisted  Recruiter 
Canvasser  Course:  Program  of  Instruction  (1984).  Note  that 

decimal  numbers  indicate  sub-objectives  for  each  main  objective. 

1.0  Explain  applicable  security  requirements  using  CNRC  security 
directives . 

1.1  Explain  NRS  security  requirements. 

1.2  Explain  test  material  security  requirements. 

1.3  Explain  security  requirements  for  Government  TRs,  bulk, 
transportation  and  meal  tickets. 

1.4  Explain  security  requirements  concerning  prospects' 
personal  information. 

2.0  Discuss  the  relationships  among  the  Recruiter  Qualification 
Standards  (RQS),  the  Freeman  Plan  and  recruiter  incentives. 

2.1  Describe  the  purpose  and  minimum  standards  for 
qualification  under  RQS. 

2.2  Describe  the  purpose,  operation  of,  and  benefits  derived 
from  the  Freeman  Plan. 

2.3  Identify  various  recruiter  incentives. 

3.0  Diagram  the  CNRC  production  chain  of  command. 

3.1  Identify  the  CNRC  production  chain  of  command. 

3.2  Using  the  CNRC  production  chain  of  command,  explain  how 
information  is  transmitted  and/or  received. 

4.0  Interpret  the  information  displayed  on  a SMART  Board. 

4.1  Identify  assigned  territory  boundaries. 

4.2  Identify  types  of  educational  institutions. 
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4.3  Identify  population  centers. 

4.4  Identify  leisure-time  activity  centers. 

4.5  Explain  the  purpose  of  specific  information  on  the  SMART 
Board . 

4.6  Discuss  the  value  of  the  SMART  Board  in  developing 

marketing  strategies  . 

5.0  Describe  the  functions  of  all  departments  within  the  NRD . 

5.1  Identify  all  departments  within  the  NRD. 

5.2  Explain  the  recruiter  support  functions  of  each 
department . 

6.0  Explain  how  community  personnel  can  contribute  to  the 

recruiting  effort. 

6.1  Identify  those  personnel  who  are  in  a position  to 
contribute  to  the  recruiting  effort. 

6.2  Describe  methods  for  presenting  awards  and  letters  of 
appreciation  to  personnel  who  contribute  to  the 
recruiting  effort. 

7.0  Using  applicable  CNRC  directives,  describe  the  purpose  and 

function  of  the  prospect  card  system. 

7.1  List  the  sources  from  which  names  of  prospects  can  be 
obtained. 

7.2  Describe  what  specific  information  must  be  transcribed 
onto  the  prospect  card. 

7.3  Explain  the  purpose  of  each  card  in  relation  to  its 

placement  in  the  various  files  of  the  system. 

7.4  Explain  how  the  prospect  card  system  operates. 

7.5  Given  specific  information  on  a prospect,  complete  a 
prospect  card. 

7.6  Given  a partially  completed  prospect  card,  update  it 
with  additional  information. 

7.7  Explain  the  procedures  for  maintaining  the  prospect  card 
system. 
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Explain 

system. 

the  procedures 

for 

purging 

the 

prospect  card 

Explain 

system. 

the  procedures 

for 

updating 

the 

prospect  card 

8.0  Given  hypothetical  prospecting  information,  develop  a 

prospecting  plan  which  includes  all  prospecting  categories. 

8.1  Document  prospecting  and  processing  activities  on 
applicable  forms. 

8.2  Identify  resources  used  to  develop  a prospecting  plan. 

8.3  Describe  how  the  listed  resources  can  be  used  to  your 
advantage . 

8.4  Given  hypothetical  prospecting  information  and  monthly 
goals,  compute  success  ratios  for  each  prospecting 
activity  necessary  to  accomplish  the  goals. 

8.5  Compile  prospecting  activity  levels  necessary  to 
accomplish  the  goals. 

8.6  Given  a map  of  a recruiting  territory  and  pertinent 
information,  develop  a travel  itinerary. 

8.7  Using  computed  prospecting  information,  schedule 
prospecting  activity  within  specific  time  frames  on  a 
daily  planner. 

8.8  Explain  the  procedures  for  implementing  the  prospecting 
plan. 

8.9  Analyze  an  existing  prospecting  plan,  determine  its 
inadequacies  and  describe  how  the  plan  could  be 
improved . 

9.0  Make  appointments  using  the  methods  of  each  prospecting 

category . 

9.1  Describe  all  prospecting  categories  without  error. 

9.2  Given  an  approved  telephone  prospecting  script  and  a 
list  of  prospects'  names  and  phone  numbers,  make 
appointments  and  seek  referrals  using  a telephone. 

9.3  Given  an  approved  prospecting  script,  describe 
appointment  power  using  the  personally  developed 
contact  (PDC)  method. 
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9.4  Make  appointments  using  pertinent  referral  information 
and  an  approved  prospecting  script. 

9.5  Discuss  the  advantages  and  disadvantages  of  prospecting 
with  mail-outs. 

9.6  Describe  what  advertising  methods  would  be  most 
effective  for  obtaining  appointments. 

9.7  Explain  how  the  Local  Effective  Accessions  Delivery 
System  (LEADS)  can  reduce  the  recruiter's  prospecting 
activity  time. 

10.0  Perform  a simulated  sales  interview  in  an  office  situation, 

achieving  75%  accuracy  based  on  the  standard  sales  interview 

checklist . 

10.1  Discuss  aspects  of  generating  and  evaluating  pre- 

approach blueprinting  techniques  and  information. 

10.2  Initiate  and  maintain  conversation  with  an  individual, 
and  establish  the  individual's  Dominant  Buying  Motive 
(DBM). 

10.3  Arouse  the  individual's  interest  and  curiosity  based 
on  his/her  DBM. 

10.4  Present  the  individual  with  specific  Units  of 
Conviction  based  on  his/her  DBM. 

10.5  Demonstrate  the  procedures  for  handling  the 

individual's  objections  during  the  interview. 

10.6  Create  desire  in  the  individual  by  verbally  painting  a 
mental  picture  based  on  the  five  natural  senses. 

10.7  Close  the  sale  using  either  the  Weighing  Close  or 
Minor  Decision  Close. 

11.0  Determine  an  individual's  qualifications  for  enlistment. 

11.1  Identify  Basic  Enlistment  Eligibility  Requirements. 

11.2  Explain  under  what  circumstances  a waiver  is  required. 

Identify  requirements  and  options  available  to 
prior-service  individuals  seeking  re-enlistment. 


11.3 
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12.0  In  a simulated  community  event,  present  a proper  Navy  image 
in  accordance  with  CNRC  and  U.S.  Navy  directives. 

12.1  Deliver  Navy  awareness  speeches. 

12.2  Present  a proper  Navy  image  at  all  times. 

13.0  Using  hypothetical  applicant  information,  complete  pre- 
enlistment kits  by  printing  legibly  in  black/blue-black 
ink.  All  kit  errors  must  be  corrected  by  the  student  until 
the  kit  is  error-free. 

13.1  Complete  a standard  pre-enlistment  kit. 

13.2  Complete  a pre-enlistment  kit  requiring  a waiver. 

13.3  Complete  a pre-enlistment  kit  for  a prior-service 
applicant . 

13.4  Quality  control  a pre-enlistment  kit. 

14.0  Using  CNRC  and  MEPCOM  directives,  explain  how  an  applicant 
is  scheduled  and  processed  through  the  Military  Enlistment 
Processing  Station  (MEPS). 


14.1 

Describe 

the  physical  testing 

procedures  at  MEPS. 

14.2 

Describe 

various  physical 

profiles  and  their 

significance . 

14.3 

Describe 

mental  testing  proced 

ures  at  MEPS. 

15.0  Using  and  complying  with  applicable  directives,  notices  and 

instructions,  demonstrate  the  ability  to  use  the  Recruiting 

Manual  and  the  RETOPS-ENL  manual  as  reference  tools. 

15.1  Maintain/update  the  Recruiting  Manual  and  RETOPS-ENL 
Manual  without  error. 

15.2  Demonstrate  proficiency  in  using  the  Recruiting  Manual 
as  a reference  tool  for  enlistment /re-enlistment 
purposes . 

15.3  Demonstrate  proficiency  in  using  the  RETOPS-ENL  Manual 
as  a reference  tool. 

16.0  Explain  how  to  instruct  and  counsel  applicants. 

16.1  Describe  DEP  responsibilities. 
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16.2  Explain  how  to  pursue  qualified-not-enlisted  (QNE) 
applicants . 

16.3  Describe  how  to  counsel  permanently  rejected 

applicants  . 

17.0  Using  required  forms  and  documents,  analyze  and  adjust 

processing  results  from  interviews  to  accessions. 

17.1  Analyze  the  results  of  previously  documented 

processing  activities. 

17.2  Explain  the  procedures  for  adjusting  processing 
activity  to  improve  results. 


Appendix  F 


CORRESPONDENCE  BETWEEN  JOB  TASRS  AND  COURSE  OBJECTIVES 


The  following  correspondence  listing  was  excerpted  from  the 
United  States  Navy  Recruiting  Command  Enlisted  Recruiter  Canvasser 
Course:  Program  of  Instruction  (1984). 


OBJECTIVE  NUMBER 

RELATED  TASR(S) 

1.1 

1.2 

1.3 

1.4 

1.1.1 

1.1.2 

1.1.3 

1.1.4 

2.1 

2.2 

2.3 

1.5.1 

1.5.2 

1.5.3 

3.1 

3.2 

1.3.1 

1.3.3 

4.1 

4.2 

4.3 

4.4 

4.5 

4.6 

2.1.1 

2.1.2 

2.1.3 

2.1.4 

3.4.5 
3.4.5 

5.1 

5.2 

2.2.1 

2.2.1 

6.1 

6.2 

2.3.1 

2.3.2,  2.3.3 

7.1 

7.2 

7.3 

7.4 

7.5 

7.6 

7.7 

7.8 

7.9 

132 

3.2.1 

3.2.2 

3.2.3 

3.2.3,  3.2.4, 
3.2.5 

3.2.2 

3.2.4 

3.2.2,  3.2.4, 
3.2.5 

3.2.4,  3.2.5 
3.2.4,  3.2.5 
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OBJECTIVE  NUMBER 

RELATED  TASK(S) 

8.1 

3. 3. 1.4,  3. 3. 2. 5,  3. 3. 3. 6, 

3. 3. 4. 5,  3. 3. 5. 6,  4.5.1,  4.5.2 

8.2 

8.3 

8.4 

8.5 

8.6 

8.7 

8.8 
8.9 

3.4.2 

3.4.3 

3.4.4 

3.4.4 

3.4.5 

3.4.6,  3.4.7 
3.4.8,  3.5 

3.6.1,  3.6.2,  3.6.3,  3.6.4 

9.2 

9.3 

9.4 

9.5 

9.6 

9.7 

3.3.1 

3.3.2 

3.3.3 

3.3.4 

3.3.5 

3.3.5,  4.5.3 

10.1 

10.2 

10.3 

10.4 

10.5 

10.6 
10.7 

4.4.1,  4.2.1 

4.1.2,  4.2.2 

4.1.3,  4.2.2 

4.1.4,  4.2.2 

4.1.4,  4.2.2 

4.1.5,  4.2.2 

4.1.6,  4.2.2 

11.1 

11.2 

11.3 

4.3.1 

4.3.2 

4.3.3 

12.1 

12.2 

4.4.1 

4.4.2 

13.1 

13.2 

13.3 

13.4 

5.1.1 

5.1.2 

5.1.3 

5. 1.1.1,  5. 1.2.1,  5. 1.3.1 

14.1 

14.2 

14.3 

5.2.3 

5.2.3,  5.3 
5.2.3 
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OBJECTIVE  NUMBER 

RELATED  TASK(S) 

15.1 

15.2 

15.3 

5.3 

5.3 

5.3 

16.1 

16.2 

16.3 

5.4.1 

5.4.2 

5.4.3,  5.4.4 

17.1 

17.2 

5.5.1,  5.5.2,  5.5.3 
5.5.4 

Appendix  G 


COMPUTATION  OF  GRADUATE  SURVEY  SAMPLE  SIZE 


This  appendix  describes  the  method  used  to  estimate  the 
sample  size  of  graduates'  surveys  needed  to  determine,  within  a 
95%  confidence  level,  whether  a clear  majority  of  a group  of 
students  advocated  a change  to  the  instructional  process.  The 
average  class  size  (group)  in  the  study  was  approximately  30 
students.  Each  group  was  assumed  to  exhibit  a normally 
distributed  population  variance  in  terras  of  its  perceptions  of 
course  adequacy.  By  invoking  the  central  limit  theorem,  the 
binomial  distribution  of  the  small  group  size  was  replaced  with  an 
approximation  to  a normal  distribution. 

The  criteria  established  by  the  school  to  define  "a  clear 
majority  of  graduates”  was  any  percentage  greater  than  or  equal  to 
50%  of  a graduating  class.  Given  this,  a one-tailed  confidence 
interval  was  constructed  around  a cutoff  value  of  0.5. 

The  notation  below  applies  to  the  following  computations: 

P = proportion  of  graduates  recommending  a change. 

C = cutoff  value  to  determine  a clear  majority  (0.5). 

N = group  size. 

R = return  rate  expected  (0.2) 
n = sample  size  of  returned  questionnaires. 

0(  = confidence  level  (.95). 

z = normal  deviate  for  a given  confidence  level. 

Q = 1-P. 

n'  = sample  size  of  questionnaires  to  be  sent  out. 
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The  formula  for  the  confidence  limits  of  a binomial  parameter 
estimate  is: 

p ± z .\JPQ/n  . 

(l— a r 

Since  the  confidence  interval  is  one-sided,  the  limit  becomes: 

p - z J PQ/n  ~>,  C. 

2\  l-(X  r 

Adding  the  discrete  binomial  correction  factor  and  the  finite 

population  correction  factor  yields: 

p - (z  J PQ/n  v/l-n/N'  + l/2n)  C 

2(l-«)v  v 

or 

P “ (Z2(  1-rt  )7PQ/n  “ PQ/N'  + 1/2n)  ^ C* 

Since  alpha  = .95  and  C = .5,  we  have: 

p - ( 1. 645 J PQ/n  - PQ/N’  + l/2n)  ^ 0.5 
For  this  study,  the  expected  proportion  recommending  a change 
to  the  curriculum  (p)  was  established  by  the  school's 
administration  as  0.8.  Thus,  Q ( =1— P)  equals  0.2.  Group  size  (N) 
was  30.  Entering  these  into  the  formula  yields: 

0.8  - ( 1 .645  yj  . 16/n  - . 16/30*  + l/2n)  0.5 

0.3  - 1/ 2n  >,  1 .645  \J  . 16/n  - .16/30’ 

0.3n/ 1 . 645  - 0.5/1.645  ^/.16n  - .0053n2‘ 

( . 1824n  - .3040)2  .16n  - .0053n2 

• 0333n2  - . 1 109n  + .0924  .16n  - .0053n2 

• 0386n2  - . 2709n  + .0924  >,  0. 

Using  the  quadratic  formula  to  solve  for  n yields: 


n (.2709  ± \J  .0734  - .0143’  )/.0772 
3.5091  ± 3.1490. 
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Thus  , 


n >y  0.3601  or  n 6.6581. 

By  selecting  the  greater  value,  it  can  be  seen  that  at  least  7 
questionnaires  (6.6581  rounded  up)  must  be  returned  to  assure  a 
majority  recommendation  with  95%  confidence.  The  final  step  is  to 
determine  how  many  surveys  to  send  out  (n')  to  course  graduates  to 
ensure  an  expected  return  rate  of  7 questionnaires  per  group. 
Since  the  estimated  return  rate  is  0.2,  we  have: 
n'  = n/R  = 6.6851/0.2 
n’  = 33.2905. 


Since  n*>N,  surveys  were  sent  to  each  graduate  of  every  class  in 
the  study. 
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